[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:31:59 dist-test-slave-n3p4 krb5kdc[396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:31:59 dist-test-slave-n3p4 krb5kdc[396](info): set up 2 sockets
Sep 27 18:31:59 dist-test-slave-n3p4 krb5kdc[396](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:32:00.142721 389 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:00 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:00.313383 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34243
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:00.642798 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:00.643090 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:00.643244 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:00.650756 409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:00.650869 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:00.650951 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:00.651057 409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:00.651124 409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:00.660888 409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:00.662257 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:00.663558 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:00.675596 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:00.675710 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:00.675711 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:00.676306 409 server_base.cc:1034] running on GCE node
I20260927 18:32:00.677042 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:00.678431 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:00.679623 409 hybrid_clock.cc:648] HybridClock initialized: now 1790533920679539 us; error 73 us; skew 500 ppm
I20260927 18:32:00.679977 409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:00 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:00.683117 409 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:00.684576 409 webserver.cc:460] Webserver started at http://127.0.97.126:37005/ using document root <none> and password file <none>
I20260927 18:32:00.685232 409 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:00.685357 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:00.685660 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:00.689303 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "0d28899900cd48d393e53ef60f3fb7b9"
format_stamp: "Formatted at 2026-09-27 18:32:00 on dist-test-slave-n3p4"
server_key: "b6176e651c7dcd7cb352727284aa47ea"
server_key_iv: "85a6254cd949017deabc786bf1a0d0ac"
server_key_version: "encryptionkey@0"
I20260927 18:32:00.690150 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "0d28899900cd48d393e53ef60f3fb7b9"
format_stamp: "Formatted at 2026-09-27 18:32:00 on dist-test-slave-n3p4"
server_key: "b6176e651c7dcd7cb352727284aa47ea"
server_key_iv: "85a6254cd949017deabc786bf1a0d0ac"
server_key_version: "encryptionkey@0"
I20260927 18:32:00.695091 409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:32:00.698851 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:00.700124 409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:32:00.700297 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "0d28899900cd48d393e53ef60f3fb7b9"
format_stamp: "Formatted at 2026-09-27 18:32:00 on dist-test-slave-n3p4"
server_key: "b6176e651c7dcd7cb352727284aa47ea"
server_key_iv: "85a6254cd949017deabc786bf1a0d0ac"
server_key_version: "encryptionkey@0"
I20260927 18:32:00.700476 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:00.716416 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:00.721151 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:00.721437 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:00.741642 409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34243
I20260927 18:32:00.741688 475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34243 every 8 connection(s)
I20260927 18:32:00.743135 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:00.746790 476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:00.749397 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 409
I20260927 18:32:00.749706 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:00.752164 389 external_mini_cluster.cc:1428] Setting key 9c3d444f3657e75699785858ae806dc0
I20260927 18:32:00.759505 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9: Bootstrap starting.
I20260927 18:32:00.762624 476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:00.763567 476 log.cc:824] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:00.766208 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9: No bootstrap required, opened a new log
I20260927 18:32:00.776014 476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } }
I20260927 18:32:00.776417 476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:00.776542 476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d28899900cd48d393e53ef60f3fb7b9, State: Initialized, Role: FOLLOWER
I20260927 18:32:00.777215 476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } }
I20260927 18:32:00.777495 476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:00.777675 476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:00.777881 476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:00.781003 476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } }
I20260927 18:32:00.781522 476 leader_election.cc:302] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d28899900cd48d393e53ef60f3fb7b9; no voters:
Sep 27 18:32:00 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:00.782203 476 leader_election.cc:288] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:00.782341 481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:00.783959 481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0d28899900cd48d393e53ef60f3fb7b9, State: Running, Role: LEADER
I20260927 18:32:00.784693 476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:00.784564 481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } }
I20260927 18:32:00.788760 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d28899900cd48d393e53ef60f3fb7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } } }
I20260927 18:32:00.789537 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:00.790077 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d28899900cd48d393e53ef60f3fb7b9. Latest consensus state: current_term: 1 leader_uuid: "0d28899900cd48d393e53ef60f3fb7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28899900cd48d393e53ef60f3fb7b9" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34243 } } }
I20260927 18:32:00.790402 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:00.799695 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:00.756760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.0.97.126:34243)
0927 18:32:00.757680 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:00.757695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:00.759136 (+ 1441us) server_negotiation.cc:407] Connection header received
0927 18:32:00.769108 (+ 9972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:00.769133 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:00.769639 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:00.770061 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:00.772885 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:00.774614 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:00.776270 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:00.777108 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:00.777308 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:00.783777 (+ 6469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:00.783789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:00.783813 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:00.783866 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:00.789152 (+ 5286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:00.790749 (+ 1597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:00.790763 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:00.790775 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:00.791109 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:00.791888 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:00.791897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:00.791904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:00.792819 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:00.793192 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:00.797094 (+ 3902us) server_negotiation.cc:299] Negotiation successful
0927 18:32:00.797754 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":408,"threads_started":1}
I20260927 18:32:00.807834 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:00.815732 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:00.825343 487 catalog_manager.cc:1347] Generated new cluster ID: b5f089e9b0f34ca58034069646afc289
I20260927 18:32:00.825487 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:00.841868 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:00.843684 487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:00.858616 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d28899900cd48d393e53ef60f3fb7b9: Generated new TSK 0
I20260927 18:32:00.859654 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:00.910646 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:01.241688 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:01.242028 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:01.242192 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:01.249671 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:01.249852 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:01.250029 503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:01.259670 503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:01.260967 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:01.262391 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:01.276311 509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:01.276324 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:01.276360 508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:01.279310 503 server_base.cc:1034] running on GCE node
I20260927 18:32:01.279920 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:01.280920 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:01.282140 503 hybrid_clock.cc:648] HybridClock initialized: now 1790533921282094 us; error 44 us; skew 500 ppm
I20260927 18:32:01.282447 503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:01 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:01.285493 503 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:01.287068 503 webserver.cc:460] Webserver started at http://127.0.97.65:39841/ using document root <none> and password file <none>
I20260927 18:32:01.287781 503 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:01.287921 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:01.288223 503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:01.291163 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "0ccc2774f53a4777b3470f3407d48a84"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "6405df510ba3fa80e02e869e5835d7bd"
server_key_iv: "ab235fca03913483e172b63eed1e130a"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.291972 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0ccc2774f53a4777b3470f3407d48a84"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "6405df510ba3fa80e02e869e5835d7bd"
server_key_iv: "ab235fca03913483e172b63eed1e130a"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.297873 503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:32:01.301195 518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.302596 503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:01.302726 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "0ccc2774f53a4777b3470f3407d48a84"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "6405df510ba3fa80e02e869e5835d7bd"
server_key_iv: "ab235fca03913483e172b63eed1e130a"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.302903 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:01.315749 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:01.319938 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:01.320186 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:01.321111 503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:01.322485 503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:01.322558 503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.322647 503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:01.322686 503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.365366 503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:35197
I20260927 18:32:01.365404 630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:35197 every 8 connection(s)
I20260927 18:32:01.366802 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:01.368477 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 503
I20260927 18:32:01.368695 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:01.369407 389 external_mini_cluster.cc:1428] Setting key 4e2ff57b2189d0aaca04acb4721ffd97
Sep 27 18:32:01 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:32:01.391268 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:01.369961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47309 (local address 127.0.97.126:34243)
0927 18:32:01.370421 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:01.370427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:01.373869 (+ 3442us) server_negotiation.cc:407] Connection header received
0927 18:32:01.374690 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:01.374696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:01.374782 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:01.374919 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:01.377079 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.377899 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.379555 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.380773 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.380915 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:01.384973 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:01.384982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:01.384987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:01.385032 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:01.388101 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:01.388875 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:01.388882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:01.388886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:01.389011 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:01.389454 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:01.389459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:01.389462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:01.389721 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:01.389899 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:01.390666 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 18:32:01.390901 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":236,"threads_started":1}
I20260927 18:32:01.392269 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:01.370285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34243 (local address 127.0.97.65:47309)
0927 18:32:01.371100 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:01.371160 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:01.373909 (+ 2749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:01.375219 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:01.375233 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:01.376056 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:01.376901 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:01.376921 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.378082 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.378087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:01.379403 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:01.379415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.381028 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.381034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:01.381263 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:01.382057 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:01.382081 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:01.384712 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:01.388271 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:01.388281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:01.388303 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:01.388747 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:01.389125 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:01.389133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:01.389136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:01.389331 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:01.389890 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:01.389898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:01.390296 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 18:32:01.390568 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 18:32:01.391129 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":379,"threads_started":1}
I20260927 18:32:01.393992 631 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34243
I20260927 18:32:01.394351 631 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:01.395110 631 heartbeater.cc:511] Master 127.0.97.126:34243 requested a full tablet report, sending...
I20260927 18:32:01.397190 441 ts_manager.cc:194] Registered new tserver with Master: 0ccc2774f53a4777b3470f3407d48a84 (127.0.97.65:35197)
I20260927 18:32:01.398680 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:47309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:01.412884 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:01.743115 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:01.743461 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:01.743599 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:01.755439 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:01.755595 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:01.755782 637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:01.769337 637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:01.770933 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:01.772529 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:01.786420 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:01.786674 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:01.786674 645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:01.787096 637 server_base.cc:1034] running on GCE node
I20260927 18:32:01.788275 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:01.789238 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:01.790449 637 hybrid_clock.cc:648] HybridClock initialized: now 1790533921790396 us; error 44 us; skew 500 ppm
I20260927 18:32:01.790755 637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:01 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:01.794139 637 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:01.795717 637 webserver.cc:460] Webserver started at http://127.0.97.66:40535/ using document root <none> and password file <none>
I20260927 18:32:01.796463 637 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:01.796609 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:01.796934 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:01.799990 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "c3a4f7a0a25b40efbd44d0ced78bc7ca"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "570ea24d1178c2b8546a22aec6afbb17"
server_key_iv: "427accd174297739da892a9285b696ef"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.800884 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "c3a4f7a0a25b40efbd44d0ced78bc7ca"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "570ea24d1178c2b8546a22aec6afbb17"
server_key_iv: "427accd174297739da892a9285b696ef"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.806186 637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:32:01.809835 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.811124 637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:01.811301 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "c3a4f7a0a25b40efbd44d0ced78bc7ca"
format_stamp: "Formatted at 2026-09-27 18:32:01 on dist-test-slave-n3p4"
server_key: "570ea24d1178c2b8546a22aec6afbb17"
server_key_iv: "427accd174297739da892a9285b696ef"
server_key_version: "encryptionkey@0"
I20260927 18:32:01.811504 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:01.836787 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:01.840936 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:01.841230 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:01.842260 637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:01.843627 637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:01.843730 637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.843879 637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:01.843950 637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:01.879213 637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38193
I20260927 18:32:01.879273 764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38193 every 8 connection(s)
I20260927 18:32:01.880558 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:01.889292 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 637
I20260927 18:32:01.889528 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:01.890153 389 external_mini_cluster.cc:1428] Setting key 7d2488673b52e8927e400884ec85913d
Sep 27 18:32:01 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:01.902902 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:01.883539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41023 (local address 127.0.97.126:34243)
0927 18:32:01.883734 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:01.883739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:01.884990 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 18:32:01.886959 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:01.886963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:01.887026 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:01.887125 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:01.888570 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.889270 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.891419 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.891955 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.892140 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:01.896128 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:01.896138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:01.896146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:01.896573 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:01.899375 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:01.900381 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:01.900386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:01.900388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:01.900522 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:01.901052 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:01.901055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:01.901057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:01.901304 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:01.901448 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:01.902443 (+ 995us) server_negotiation.cc:299] Negotiation successful
0927 18:32:01.902612 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":76}
I20260927 18:32:01.904609 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:01.884035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34243 (local address 127.0.97.66:41023)
0927 18:32:01.884819 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:01.884864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:01.886699 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:01.887328 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:01.887337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:01.887843 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:01.888414 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:01.888432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.889431 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.889438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:01.891249 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:01.891272 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:01.892243 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:01.892251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:01.892516 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:01.893245 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:01.893268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:01.895875 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:01.899519 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:01.899532 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:01.899553 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:01.900226 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:01.900658 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:01.900667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:01.900674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:01.900932 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:01.901470 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:01.901482 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:01.901982 (+ 500us) client_negotiation.cc:769] Sending connection context
0927 18:32:01.902336 (+ 354us) client_negotiation.cc:240] Negotiation successful
0927 18:32:01.902955 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":272,"threads_started":1}
I20260927 18:32:01.906456 765 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34243
I20260927 18:32:01.906737 765 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:01.907497 765 heartbeater.cc:511] Master 127.0.97.126:34243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:01.908964 441 ts_manager.cc:194] Registered new tserver with Master: c3a4f7a0a25b40efbd44d0ced78bc7ca (127.0.97.66:38193)
I20260927 18:32:01.909996 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:41023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:01.933929 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:02.261996 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:02.262319 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:02.262497 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:02.269659 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:02.269858 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:02.270053 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:02.280243 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:02.281553 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:02.283025 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:02.296844 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:02.296857 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:02.297485 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:02.297816 770 server_base.cc:1034] running on GCE node
I20260927 18:32:02.298369 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:02.299261 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:02.300460 770 hybrid_clock.cc:648] HybridClock initialized: now 1790533922300402 us; error 55 us; skew 500 ppm
I20260927 18:32:02.300726 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:02.303866 770 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:02.305325 770 webserver.cc:460] Webserver started at http://127.0.97.67:38373/ using document root <none> and password file <none>
I20260927 18:32:02.306129 770 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:02.306246 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:02.306607 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:02.309903 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "2d873df8cb8240feb280c404d56fea60"
format_stamp: "Formatted at 2026-09-27 18:32:02 on dist-test-slave-n3p4"
server_key: "403d47770c183704ebec659d6deed26d"
server_key_iv: "ffc5c4442bd08469e7f2985da57073df"
server_key_version: "encryptionkey@0"
I20260927 18:32:02.310683 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "2d873df8cb8240feb280c404d56fea60"
format_stamp: "Formatted at 2026-09-27 18:32:02 on dist-test-slave-n3p4"
server_key: "403d47770c183704ebec659d6deed26d"
server_key_iv: "ffc5c4442bd08469e7f2985da57073df"
server_key_version: "encryptionkey@0"
I20260927 18:32:02.315791 770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:02.319231 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:02.320611 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:32:02.320811 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "2d873df8cb8240feb280c404d56fea60"
format_stamp: "Formatted at 2026-09-27 18:32:02 on dist-test-slave-n3p4"
server_key: "403d47770c183704ebec659d6deed26d"
server_key_iv: "ffc5c4442bd08469e7f2985da57073df"
server_key_version: "encryptionkey@0"
I20260927 18:32:02.321007 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:02.352097 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:02.357000 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:02.357276 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:02.358273 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:02.359599 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:02.359678 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:02.359767 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:02.359827 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:02.398233 770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:34699
I20260927 18:32:02.398341 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:34699 every 8 connection(s)
I20260927 18:32:02.399669 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:02.401110 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 770
I20260927 18:32:02.401324 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:02.401809 389 external_mini_cluster.cc:1428] Setting key 6a176d5d26321d2ec1c64fb747c4f847
I20260927 18:32:02.402061 631 heartbeater.cc:503] Master 127.0.97.126:34243 was elected leader, sending a full tablet report...
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:02.427017 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.403145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57385 (local address 127.0.97.126:34243)
0927 18:32:02.403357 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.403364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.405968 (+ 2604us) server_negotiation.cc:407] Connection header received
0927 18:32:02.408040 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.408045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.408124 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.408242 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.409659 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.410839 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.412340 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.412748 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.412880 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.420888 (+ 8008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.420894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.420898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.420934 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.423707 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.424561 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.424573 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.424575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.424733 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.425160 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.425166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.425169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.425471 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.425655 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.426443 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.426672 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:02.428023 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.403131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34243 (local address 127.0.97.67:57385)
0927 18:32:02.405278 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:02.405872 (+ 594us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:02.407801 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:02.408469 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:02.408477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:02.408955 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:02.409492 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:02.409509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.410986 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.410992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:02.412201 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:02.412212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.412957 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.412966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:02.413211 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.417537 (+ 4326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:02.417577 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:02.420647 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:02.423869 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:02.423879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:02.423903 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:02.424425 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:02.424844 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:02.424850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:02.424854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:02.425037 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:02.425621 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:02.425630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:02.426058 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 18:32:02.426357 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 18:32:02.426899 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"thread_start_us":1599,"threads_started":1}
I20260927 18:32:02.429466 898 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34243
I20260927 18:32:02.429898 898 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:02.430851 898 heartbeater.cc:511] Master 127.0.97.126:34243 requested a full tablet report, sending...
I20260927 18:32:02.432480 440 ts_manager.cc:194] Registered new tserver with Master: 2d873df8cb8240feb280c404d56fea60 (127.0.97.67:34699)
I20260927 18:32:02.433332 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:57385
I20260927 18:32:02.444468 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:02.527109 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.510224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.97.65:35197)
0927 18:32:02.511637 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.511649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.511684 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:32:02.511766 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.511778 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.512053 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.512236 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.513610 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.515134 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.516715 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.517359 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.517462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.521465 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.521476 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.521491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.521529 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.524408 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.525004 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.525011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.525013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.525143 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.525645 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.525649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.525651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.526026 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.526259 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.526503 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.526757 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":1072,"threads_started":1}
W20260927 18:32:02.528369 546 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40096
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:02.631105 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.614943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.97.65:35197)
0927 18:32:02.615655 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.615660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.615678 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:32:02.615776 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.615780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.615843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.615932 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.618378 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.619127 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.620451 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.620878 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.620986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.626018 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.626024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.626027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.626065 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.628599 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.629365 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.629369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.629372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.629498 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.629913 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.629917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.629919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.630190 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.630370 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.630723 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.630880 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543}
I20260927 18:32:02.632654 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 503
I20260927 18:32:02.646343 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.633826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.0.97.65:35197)
0927 18:32:02.634019 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.634025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.634197 (+ 172us) server_negotiation.cc:407] Connection header received
0927 18:32:02.634381 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.634393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.634445 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.634539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.635697 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.636386 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.637595 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.638054 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.638184 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.640578 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.640584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.640587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.640618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.643589 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.644250 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.644258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.644262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.644410 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.644804 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.644812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.644815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.645090 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.645347 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.645697 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.645949 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:32:02.647116 626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:02.736603 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 503
I20260927 18:32:02.757249 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 637
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:02.774318 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.758401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.0.97.66:38193)
0927 18:32:02.759016 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.759026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.759195 (+ 169us) server_negotiation.cc:407] Connection header received
0927 18:32:02.759432 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.759445 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.759767 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.759959 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.761230 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.762473 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.763769 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.764341 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.764506 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.768692 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.768704 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.768718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.768750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.771467 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.772103 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.772110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.772112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.772224 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.772607 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.772612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.772615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.772972 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.773280 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.773632 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.773968 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
I20260927 18:32:02.775523 760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:02.857688 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 637
I20260927 18:32:02.879822 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 770
Sep 27 18:32:02 dist-test-slave-n3p4 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:02.896030 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:02.880921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.0.97.67:34699)
0927 18:32:02.881487 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:02.881494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:02.881524 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:32:02.881592 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:02.881601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:02.881882 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:02.882058 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:02.883176 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.884245 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.885835 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:02.886533 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:02.886639 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:02.890751 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:02.890762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:02.890776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:02.890808 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:02.893523 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.894306 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.894313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.894315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.894429 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:02.894742 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:02.894745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:02.894747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:02.895095 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:02.895309 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:02.895550 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:32:02.895758 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
I20260927 18:32:02.897315 893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:02.987205 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 770
I20260927 18:32:03.003595 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 409
I20260927 18:32:03.004751 471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:03.112609 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 409
2026-09-27T18:32:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4064 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:05 dist-test-slave-n3p4 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:05 dist-test-slave-n3p4 krb5kdc[919](info): set up 2 sockets
Sep 27 18:32:05 dist-test-slave-n3p4 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260927 18:32:06.152750 389 mini_kdc.cc:121] Time spent starting KDC: real 2.999s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 18:32:06.289990 389 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.137s user 0.003s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 18:32:06.394838 389 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.105s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 18:32:06.540582 389 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.146s user 0.001s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/test-user.keytab.
W20260927 18:32:06.885843 389 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.345s user 0.000s sys 0.004s
Sep 27 18:32:06 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
W20260927 18:32:07.737124 389 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.126: real 0.806s user 0.000s sys 0.008s
I20260927 18:32:07.738269 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45799
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:08.092226 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:08.092578 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:08.092758 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:08.100090 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:08.100244 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:08.100353 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:08.100452 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:08.100538 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:08.110616 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:08.112116 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:08.113464 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:08.125917 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:08.125859 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:08.125895 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:08.126754 932 server_base.cc:1034] running on GCE node
I20260927 18:32:08.127595 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:08.128988 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:08.130216 932 hybrid_clock.cc:648] HybridClock initialized: now 1790533928130178 us; error 54 us; skew 500 ppm
I20260927 18:32:08.130568 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:08 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:08.133708 932 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:08.135185 932 webserver.cc:460] Webserver started at http://127.0.97.126:39009/ using document root <none> and password file <none>
I20260927 18:32:08.135880 932 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:08.136034 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:08.136343 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:08.139416 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "d9f53e0729c44d41876636a7bdb0e99f"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "622da8bb424ab488638b2e9f26c818ac"
server_key_iv: "319d77445983f42e52f78bd024ff53d0"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.140204 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "d9f53e0729c44d41876636a7bdb0e99f"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "622da8bb424ab488638b2e9f26c818ac"
server_key_iv: "319d77445983f42e52f78bd024ff53d0"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.145573 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:32:08.149586 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:08.151087 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:08.151311 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "d9f53e0729c44d41876636a7bdb0e99f"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "622da8bb424ab488638b2e9f26c818ac"
server_key_iv: "319d77445983f42e52f78bd024ff53d0"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.151526 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:08.165942 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:08.170704 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:08.171031 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:08.190019 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45799
I20260927 18:32:08.190052 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45799 every 8 connection(s)
I20260927 18:32:08.191469 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:08.195468 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:08.196522 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 932
I20260927 18:32:08.196772 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:08.197454 389 external_mini_cluster.cc:1428] Setting key 4807829168609ea249a104b50ce23286
I20260927 18:32:08.208321 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f: Bootstrap starting.
Sep 27 18:32:08 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:08.211521 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:08.212616 999 log.cc:824] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:08.215598 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f: No bootstrap required, opened a new log
I20260927 18:32:08.219666 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:08.199668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.0.97.126:45799)
0927 18:32:08.200524 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:08.200536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:08.200581 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:32:08.202242 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:08.202269 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:08.202699 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:08.203109 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:08.204595 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.205995 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.207245 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.207895 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.208092 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:08.212201 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:08.212209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:08.212225 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:08.212259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:08.215616 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:08.216288 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:08.216300 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:08.216306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:08.216466 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:08.216889 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:08.216897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:08.216900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:08.217412 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:08.217738 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:08.218259 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 18:32:08.218659 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":405,"threads_started":1}
I20260927 18:32:08.223982 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } }
I20260927 18:32:08.224359 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:08.224474 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f53e0729c44d41876636a7bdb0e99f, State: Initialized, Role: FOLLOWER
I20260927 18:32:08.225065 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } }
I20260927 18:32:08.225308 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:08.225453 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:08.225621 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:08.227794 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } }
I20260927 18:32:08.228291 999 leader_election.cc:302] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f53e0729c44d41876636a7bdb0e99f; no voters:
I20260927 18:32:08.228991 999 leader_election.cc:288] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:08.229096 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:08.229810 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [term 1 LEADER]: Becoming Leader. State: Replica: d9f53e0729c44d41876636a7bdb0e99f, State: Running, Role: LEADER
I20260927 18:32:08.230387 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } }
I20260927 18:32:08.231526 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:08.233146 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f53e0729c44d41876636a7bdb0e99f. Latest consensus state: current_term: 1 leader_uuid: "d9f53e0729c44d41876636a7bdb0e99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } } }
I20260927 18:32:08.233485 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:08.234138 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f53e0729c44d41876636a7bdb0e99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f53e0729c44d41876636a7bdb0e99f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45799 } } }
I20260927 18:32:08.234567 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:08.238134 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:08.244465 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:08.254261 1009 catalog_manager.cc:1347] Generated new cluster ID: 2b9a02d17c8645479baabb7cf16bd5a7
I20260927 18:32:08.254379 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:08.278781 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:08.280206 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:08.287473 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f53e0729c44d41876636a7bdb0e99f: Generated new TSK 0
I20260927 18:32:08.288563 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:08.346503 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:08.671545 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:08.671823 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:08.671986 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:08.679363 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:08.679468 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:08.679665 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:08.689237 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:08.690562 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:08.691891 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:08.704994 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:08.705039 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:08.705016 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:08.706141 1025 server_base.cc:1034] running on GCE node
I20260927 18:32:08.706749 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:08.707674 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:08.708909 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790533928708853 us; error 59 us; skew 500 ppm
I20260927 18:32:08.709237 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:08 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:08.712512 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:08.714161 1025 webserver.cc:460] Webserver started at http://127.0.97.65:38107/ using document root <none> and password file <none>
I20260927 18:32:08.714844 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:08.714987 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:08.715301 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:08.718255 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "b8b224c56ce947fa962ffcfb467f63ab"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "3f628cdf39429a3bb10917db7644295d"
server_key_iv: "30450ace48253118fa47419011233774"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.719029 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "b8b224c56ce947fa962ffcfb467f63ab"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "3f628cdf39429a3bb10917db7644295d"
server_key_iv: "30450ace48253118fa47419011233774"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.724321 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:32:08.727885 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:08.729209 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:08.729400 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "b8b224c56ce947fa962ffcfb467f63ab"
format_stamp: "Formatted at 2026-09-27 18:32:08 on dist-test-slave-n3p4"
server_key: "3f628cdf39429a3bb10917db7644295d"
server_key_iv: "30450ace48253118fa47419011233774"
server_key_version: "encryptionkey@0"
I20260927 18:32:08.729601 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:08.749502 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:08.753831 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:08.754091 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:08.755010 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:08.756326 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:08.756423 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:08.756608 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:08.756678 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:08.790481 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38907
I20260927 18:32:08.790572 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38907 every 8 connection(s)
I20260927 18:32:08.791782 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:08.793505 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1025
I20260927 18:32:08.793687 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:08.794342 389 external_mini_cluster.cc:1428] Setting key 1548a6f51368b0119b233df15c6e0377
Sep 27 18:32:08 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:32:08.813858 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:08.794548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33283 (local address 127.0.97.126:45799)
0927 18:32:08.794999 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:08.795005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:08.796519 (+ 1514us) server_negotiation.cc:407] Connection header received
0927 18:32:08.798317 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:08.798326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:08.798415 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:08.798538 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:08.800026 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.801238 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.802846 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.803605 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.803745 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:08.807111 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:08.807121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:08.807128 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:08.807176 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:08.810618 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:08.811411 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:08.811415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:08.811418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:08.811538 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:08.811983 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:08.811988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:08.811990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:08.812242 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:08.812400 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:08.813181 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 18:32:08.813445 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260927 18:32:08.814796 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:08.795273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45799 (local address 127.0.97.65:33283)
0927 18:32:08.796353 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:08.796400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:08.798052 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:08.798742 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:08.798749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:08.799251 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:08.799788 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:08.799834 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.801404 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.801408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:08.802684 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:08.802695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:08.803816 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:08.803821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:08.804020 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:08.804693 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:08.804718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:08.806880 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:08.810759 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:08.810767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:08.810780 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:08.811272 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:08.811637 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:08.811640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:08.811643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:08.811854 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:08.812371 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:08.812380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:08.812757 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 18:32:08.813070 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:32:08.813597 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":330,"threads_started":1}
I20260927 18:32:08.816392 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45799
I20260927 18:32:08.816687 1153 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:08.817428 1153 heartbeater.cc:511] Master 127.0.97.126:45799 requested a full tablet report, sending...
I20260927 18:32:08.819778 964 ts_manager.cc:194] Registered new tserver with Master: b8b224c56ce947fa962ffcfb467f63ab (127.0.97.65:38907)
I20260927 18:32:08.821238 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:33283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:08.837298 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:09.160566 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:09.160866 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:09.160965 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:09.168257 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:09.168373 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:09.168490 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:09.178363 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:09.179649 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:09.180909 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:09.194777 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:09.194798 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:09.195796 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:09.195869 1159 server_base.cc:1034] running on GCE node
I20260927 18:32:09.196623 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:09.197575 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:09.198819 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790533929198756 us; error 53 us; skew 500 ppm
I20260927 18:32:09.199149 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:09 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:09.202461 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:09.204113 1159 webserver.cc:460] Webserver started at http://127.0.97.66:37491/ using document root <none> and password file <none>
I20260927 18:32:09.204799 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:09.204926 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:09.205235 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:09.208139 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "074b730b300344f4b6f29f28fe018ebb"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "4c1796ce3355345d9b971caf2378b8ae"
server_key_iv: "92cf29964c412a7e287094f4e14bd227"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.208922 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "074b730b300344f4b6f29f28fe018ebb"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "4c1796ce3355345d9b971caf2378b8ae"
server_key_iv: "92cf29964c412a7e287094f4e14bd227"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.214135 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 18:32:09.217474 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.218931 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:09.219110 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "074b730b300344f4b6f29f28fe018ebb"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "4c1796ce3355345d9b971caf2378b8ae"
server_key_iv: "92cf29964c412a7e287094f4e14bd227"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.219306 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:09.232455 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:09.236523 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:09.236821 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:09.237815 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:09.239210 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:09.239290 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.239449 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:09.239501 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.273734 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38409
I20260927 18:32:09.273828 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38409 every 8 connection(s)
I20260927 18:32:09.275060 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:09.285130 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1159
I20260927 18:32:09.285310 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:09.285763 389 external_mini_cluster.cc:1428] Setting key 663dbce4197f1e77b1bd368509529284
Sep 27 18:32:09 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:09.295739 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.277672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:49517 (local address 127.0.97.126:45799)
0927 18:32:09.277879 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.277884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.278961 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 18:32:09.280770 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.280776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.280866 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.280979 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:09.282510 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.283227 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.284811 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.285260 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.285408 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.289596 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:09.289601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:09.289605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:09.289638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:09.292338 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.293093 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.293099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.293102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.293233 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.293702 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.293706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.293710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.294050 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:09.294207 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.295137 (+ 930us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.295374 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:09.296883 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.278103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45799 (local address 127.0.97.66:49517)
0927 18:32:09.278794 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:09.278837 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:09.280502 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:09.281208 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:09.281217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:09.281716 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:09.282343 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.282360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.283376 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.283384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.284611 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.284622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.285484 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.285490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.285733 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.286615 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:09.286644 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:09.289367 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:09.292492 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:09.292501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:09.292520 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:09.292958 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:09.293371 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:09.293377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:09.293380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:09.293581 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:09.294199 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:09.294208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:09.294735 (+ 527us) client_negotiation.cc:769] Sending connection context
0927 18:32:09.295051 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 18:32:09.295713 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":263,"threads_started":1}
I20260927 18:32:09.298470 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45799
I20260927 18:32:09.298854 1287 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:09.299619 1287 heartbeater.cc:511] Master 127.0.97.126:45799 requested a full tablet report, sending...
I20260927 18:32:09.301118 964 ts_manager.cc:194] Registered new tserver with Master: 074b730b300344f4b6f29f28fe018ebb (127.0.97.66:38409)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:09.302376 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:49517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:09.326547 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:09.659780 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:09.660070 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:09.660262 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:09.667827 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:09.667951 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:09.668145 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:09.678040 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:09.679351 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:09.680711 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:09.693830 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:09.693874 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:09.693899 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:09.694974 1292 server_base.cc:1034] running on GCE node
I20260927 18:32:09.695593 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:09.696480 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:09.697682 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790533929697646 us; error 28 us; skew 500 ppm
I20260927 18:32:09.698009 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:09 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:09.701018 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:09.702484 1292 webserver.cc:460] Webserver started at http://127.0.97.67:43295/ using document root <none> and password file <none>
I20260927 18:32:09.703157 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:09.703297 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:09.703624 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:09.706544 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "65784f8a267c4d90b643c08c7b0cfc15"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "22b06561aee2378548674483e33cca10"
server_key_iv: "0623daabbaddc56a2adc3c63431aa560"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.707332 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "65784f8a267c4d90b643c08c7b0cfc15"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "22b06561aee2378548674483e33cca10"
server_key_iv: "0623daabbaddc56a2adc3c63431aa560"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.712561 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:09.715992 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.717617 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 18:32:09.717872 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "65784f8a267c4d90b643c08c7b0cfc15"
format_stamp: "Formatted at 2026-09-27 18:32:09 on dist-test-slave-n3p4"
server_key: "22b06561aee2378548674483e33cca10"
server_key_iv: "0623daabbaddc56a2adc3c63431aa560"
server_key_version: "encryptionkey@0"
I20260927 18:32:09.718124 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:09.745069 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:09.749249 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:09.749558 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:09.750559 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:09.751910 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:09.752014 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.752167 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:09.752240 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:09.783669 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:42337
I20260927 18:32:09.783679 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:42337 every 8 connection(s)
I20260927 18:32:09.784993 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:09.794231 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1292
I20260927 18:32:09.794405 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:09.794938 389 external_mini_cluster.cc:1428] Setting key 089a4f4b84c81daf624d6ea9c916e03a
Sep 27 18:32:09 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:09.805938 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.788034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42855 (local address 127.0.97.126:45799)
0927 18:32:09.788218 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.788222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.789424 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 18:32:09.791262 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.791267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.791384 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.791496 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:09.792926 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.793661 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.795858 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.796415 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.796547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.800040 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:09.800045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:09.800049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:09.800078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:09.803000 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.803705 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.803710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.803714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.803843 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.804259 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.804264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.804272 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.804530 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:09.804694 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.805354 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.805559 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:09.806876 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.788506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45799 (local address 127.0.97.67:42855)
0927 18:32:09.789212 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:09.789258 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:09.790972 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:09.791724 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:09.791732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:09.792219 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:09.792759 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.792775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.793830 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.793836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.795703 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.795715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.796639 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.796646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.796880 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.797550 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:09.797573 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:09.799848 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:09.803136 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:09.803146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:09.803171 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:09.803573 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:09.803951 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:09.803956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:09.803960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:09.804134 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:09.804663 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:09.804671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:09.804993 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 18:32:09.805259 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 18:32:09.805796 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":289,"threads_started":1}
I20260927 18:32:09.808159 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45799
I20260927 18:32:09.808493 1420 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:09.809338 1420 heartbeater.cc:511] Master 127.0.97.126:45799 requested a full tablet report, sending...
I20260927 18:32:09.810874 964 ts_manager.cc:194] Registered new tserver with Master: 65784f8a267c4d90b643c08c7b0cfc15 (127.0.97.67:42337)
I20260927 18:32:09.811672 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:42855
I20260927 18:32:09.822741 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:09.824419 1153 heartbeater.cc:503] Master 127.0.97.126:45799 was elected leader, sending a full tablet report...
I20260927 18:32:09.843153 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.829902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.0.97.126:45799)
0927 18:32:09.830061 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.830065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.830620 (+ 555us) server_negotiation.cc:407] Connection header received
0927 18:32:09.830759 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.830763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.830824 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.830918 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:09.832395 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.833226 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.834591 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.834971 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.835066 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.837973 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:09.837978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:09.837981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:09.838016 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:09.840898 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.841510 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.841514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.841516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.841639 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:09.842021 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:09.842024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:09.842026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:09.842357 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:09.842528 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.842772 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.842932 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:32:09.849579 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:09.856518 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:09.891656 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.869666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.0.97.65:38907)
0927 18:32:09.881198 (+ 11532us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.881210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.882183 (+ 973us) server_negotiation.cc:407] Connection header received
0927 18:32:09.882387 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.882395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.882601 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.882800 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:09.884050 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.885581 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.888954 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.890439 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.890578 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.890819 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.891157 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.891318 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11340,"thread_start_us":11169,"threads_started":1}
I20260927 18:32:09.891980 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.868923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38409 (local address 127.0.0.1:53464)
0927 18:32:09.881630 (+ 12707us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:09.881676 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:09.881920 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:09.882812 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:09.882821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:09.882849 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:09.883211 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.883221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.885167 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.885172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.889014 (+ 3842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.889030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.891030 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.891035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.891273 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.891305 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:32:09.891398 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:32:09.891543 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12444,"mutex_wait_us":2,"thread_start_us":12323,"threads_started":1}
I20260927 18:32:09.891769 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.868962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38907 (local address 127.0.0.1:58332)
0927 18:32:09.882049 (+ 13087us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:09.882066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:09.882216 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:09.883405 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:09.883409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:09.883437 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:09.883843 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.883856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.885746 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.885750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.888838 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.888851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.890656 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.890663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.890875 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.890927 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:32:09.891088 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:32:09.891287 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12969,"mutex_wait_us":67,"thread_start_us":13350,"threads_started":1}
I20260927 18:32:09.892863 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.869165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.97.66:38409)
0927 18:32:09.881402 (+ 12237us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.881413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.882153 (+ 740us) server_negotiation.cc:407] Connection header received
0927 18:32:09.882269 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.882276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.882576 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.882809 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:09.883373 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.885001 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.889177 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.890615 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.890725 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.890894 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.892423 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.892600 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12050,"thread_start_us":12257,"threads_started":1}
I20260927 18:32:09.895116 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.883236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:42337 (local address 127.0.0.1:50322)
0927 18:32:09.887243 (+ 4007us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:09.887261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:09.887393 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:09.888257 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:09.888261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:09.888284 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:09.888701 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.888712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.890166 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.890171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.892380 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:09.892394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.894306 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.894313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:09.894475 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.894505 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:32:09.894638 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:32:09.894758 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3858,"mutex_wait_us":69,"thread_start_us":3720,"threads_started":1}
I20260927 18:32:09.895161 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:09.882555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.0.97.67:42337)
0927 18:32:09.886609 (+ 4054us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:09.886619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:09.887650 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 18:32:09.887777 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:09.887786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:09.888057 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:09.888262 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:09.888890 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.890027 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.892573 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:09.894109 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:09.894252 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:09.894493 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:09.894677 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 18:32:09.894826 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3877,"thread_start_us":3678,"threads_started":1}
I20260927 18:32:09.896358 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 70fdd2db24594e16ba6df2dbcd318e25 (DEFAULT_TABLE table=test-table [id=825e019cf5f041b0b7f06c2dd2c74331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:09.896366 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 70fdd2db24594e16ba6df2dbcd318e25 (DEFAULT_TABLE table=test-table [id=825e019cf5f041b0b7f06c2dd2c74331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:09.897882 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fdd2db24594e16ba6df2dbcd318e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:09.897879 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fdd2db24594e16ba6df2dbcd318e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:09.898836 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 70fdd2db24594e16ba6df2dbcd318e25 (DEFAULT_TABLE table=test-table [id=825e019cf5f041b0b7f06c2dd2c74331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:09.900187 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fdd2db24594e16ba6df2dbcd318e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:09.911408 1438 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Bootstrap starting.
I20260927 18:32:09.914489 1439 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Bootstrap starting.
I20260927 18:32:09.915354 1438 tablet_bootstrap.cc:654] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:09.917218 1438 log.cc:824] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:09.918512 1439 tablet_bootstrap.cc:654] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:09.918673 1440 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Bootstrap starting.
I20260927 18:32:09.919703 1439 log.cc:824] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:09.920747 1438 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: No bootstrap required, opened a new log
I20260927 18:32:09.921011 1438 ts_tablet_manager.cc:1397] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260927 18:32:09.923556 1440 tablet_bootstrap.cc:654] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:09.925115 1439 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: No bootstrap required, opened a new log
I20260927 18:32:09.925210 1440 log.cc:824] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:09.925390 1439 ts_tablet_manager.cc:1397] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260927 18:32:09.928622 1440 tablet_bootstrap.cc:492] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: No bootstrap required, opened a new log
I20260927 18:32:09.928877 1440 ts_tablet_manager.cc:1397] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:32:09.930117 1438 raft_consensus.cc:348] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.930474 1438 raft_consensus.cc:374] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:09.930580 1438 raft_consensus.cc:729] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65784f8a267c4d90b643c08c7b0cfc15, State: Initialized, Role: FOLLOWER
I20260927 18:32:09.931212 1438 consensus_queue.cc:260] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.936038 1439 raft_consensus.cc:348] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.936463 1439 raft_consensus.cc:374] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:09.936563 1439 raft_consensus.cc:729] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b224c56ce947fa962ffcfb467f63ab, State: Initialized, Role: FOLLOWER
I20260927 18:32:09.937175 1439 consensus_queue.cc:260] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.939141 1439 ts_tablet_manager.cc:1428] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 18:32:09.940230 1440 raft_consensus.cc:348] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.940665 1440 raft_consensus.cc:374] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:09.940764 1440 raft_consensus.cc:729] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074b730b300344f4b6f29f28fe018ebb, State: Initialized, Role: FOLLOWER
I20260927 18:32:09.941344 1440 consensus_queue.cc:260] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:09.942538 1287 heartbeater.cc:503] Master 127.0.97.126:45799 was elected leader, sending a full tablet report...
I20260927 18:32:09.942921 1420 heartbeater.cc:503] Master 127.0.97.126:45799 was elected leader, sending a full tablet report...
I20260927 18:32:09.944018 1440 ts_tablet_manager.cc:1428] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260927 18:32:09.945375 1438 ts_tablet_manager.cc:1428] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Time spent starting tablet: real 0.024s user 0.013s sys 0.011s
W20260927 18:32:10.026986 1288 tablet.cc:2369] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:32:10.037223 1421 tablet.cc:2369] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:32:10.044222 1154 tablet.cc:2369] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:10.053087 1446 raft_consensus.cc:482] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:10.053426 1446 raft_consensus.cc:504] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:10.055339 1446 leader_election.cc:288] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8b224c56ce947fa962ffcfb467f63ab (127.0.97.65:38907), 65784f8a267c4d90b643c08c7b0cfc15 (127.0.97.67:42337)
I20260927 18:32:10.063400 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.055845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34955 (local address 127.0.97.65:38907)
0927 18:32:10.056065 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.056071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.056996 (+ 925us) server_negotiation.cc:407] Connection header received
0927 18:32:10.057230 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.057242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.057308 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.057416 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:10.058203 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.058946 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.061457 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.062332 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.062438 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.062513 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.063007 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.063145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:32:10.063390 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.055819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58463 (local address 127.0.97.67:42337)
0927 18:32:10.056065 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.056076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.057037 (+ 961us) server_negotiation.cc:407] Connection header received
0927 18:32:10.057240 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.057244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.057302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.057402 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:10.058078 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.058784 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.061540 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.062490 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.062648 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.062772 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.062977 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.063110 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:32:10.063606 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.055839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38907 (local address 127.0.97.66:34955)
0927 18:32:10.056856 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:10.056873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:10.057072 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:10.057429 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:10.057434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:10.057458 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:10.058025 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:10.058036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.059115 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.059121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:10.061251 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:10.061264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.062559 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.062569 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:10.062785 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.062831 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:32:10.062958 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 18:32:10.063097 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":57,"thread_start_us":575,"threads_started":1}
I20260927 18:32:10.063614 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.055839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:42337 (local address 127.0.97.66:58463)
0927 18:32:10.056917 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:10.056932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:10.057091 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:10.057415 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:10.057420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:10.057442 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:10.057907 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:10.057917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.058916 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.058924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:10.061298 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:10.061313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.062705 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.062712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:10.062819 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.062841 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:32:10.062913 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:32:10.063066 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":255,"thread_start_us":520,"threads_started":1}
I20260927 18:32:10.064735 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fdd2db24594e16ba6df2dbcd318e25" candidate_uuid: "074b730b300344f4b6f29f28fe018ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b224c56ce947fa962ffcfb467f63ab" is_pre_election: true
I20260927 18:32:10.064881 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fdd2db24594e16ba6df2dbcd318e25" candidate_uuid: "074b730b300344f4b6f29f28fe018ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65784f8a267c4d90b643c08c7b0cfc15" is_pre_election: true
I20260927 18:32:10.065354 1108 raft_consensus.cc:2454] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074b730b300344f4b6f29f28fe018ebb in term 0.
I20260927 18:32:10.065354 1375 raft_consensus.cc:2454] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074b730b300344f4b6f29f28fe018ebb in term 0.
I20260927 18:32:10.066102 1176 leader_election.cc:302] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b730b300344f4b6f29f28fe018ebb, 65784f8a267c4d90b643c08c7b0cfc15; no voters:
I20260927 18:32:10.066573 1446 raft_consensus.cc:2781] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:10.066740 1446 raft_consensus.cc:482] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:10.066847 1446 raft_consensus.cc:3037] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:10.069001 1446 raft_consensus.cc:504] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:10.069669 1446 leader_election.cc:288] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [CANDIDATE]: Term 1 election: Requested vote from peers b8b224c56ce947fa962ffcfb467f63ab (127.0.97.65:38907), 65784f8a267c4d90b643c08c7b0cfc15 (127.0.97.67:42337)
I20260927 18:32:10.070200 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fdd2db24594e16ba6df2dbcd318e25" candidate_uuid: "074b730b300344f4b6f29f28fe018ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65784f8a267c4d90b643c08c7b0cfc15"
I20260927 18:32:10.070204 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fdd2db24594e16ba6df2dbcd318e25" candidate_uuid: "074b730b300344f4b6f29f28fe018ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b224c56ce947fa962ffcfb467f63ab"
I20260927 18:32:10.070411 1375 raft_consensus.cc:3037] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:10.070428 1108 raft_consensus.cc:3037] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:10.072934 1108 raft_consensus.cc:2454] T 70fdd2db24594e16ba6df2dbcd318e25 P b8b224c56ce947fa962ffcfb467f63ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074b730b300344f4b6f29f28fe018ebb in term 1.
I20260927 18:32:10.072958 1375 raft_consensus.cc:2454] T 70fdd2db24594e16ba6df2dbcd318e25 P 65784f8a267c4d90b643c08c7b0cfc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074b730b300344f4b6f29f28fe018ebb in term 1.
I20260927 18:32:10.073393 1177 leader_election.cc:302] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b730b300344f4b6f29f28fe018ebb, b8b224c56ce947fa962ffcfb467f63ab; no voters:
I20260927 18:32:10.073710 1446 raft_consensus.cc:2781] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:10.074155 1446 raft_consensus.cc:686] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [term 1 LEADER]: Becoming Leader. State: Replica: 074b730b300344f4b6f29f28fe018ebb, State: Running, Role: LEADER
I20260927 18:32:10.074697 1446 consensus_queue.cc:237] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } }
I20260927 18:32:10.079198 962 catalog_manager.cc:5495] T 70fdd2db24594e16ba6df2dbcd318e25 P 074b730b300344f4b6f29f28fe018ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 074b730b300344f4b6f29f28fe018ebb (127.0.97.66). New cstate: current_term: 1 leader_uuid: "074b730b300344f4b6f29f28fe018ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b224c56ce947fa962ffcfb467f63ab" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074b730b300344f4b6f29f28fe018ebb" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65784f8a267c4d90b643c08c7b0cfc15" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42337 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:10.126096 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.109671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.0.97.65:38907)
0927 18:32:10.110088 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.110097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.110425 (+ 328us) server_negotiation.cc:407] Connection header received
0927 18:32:10.110658 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.110669 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.110747 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.110864 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.112491 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.113313 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.114747 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.115624 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.115788 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.119909 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.119926 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.119942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.119977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.123464 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.124184 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.124192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.124194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.124361 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.124729 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.124732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.124734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.125147 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.125377 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.125606 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.125820 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:10.174201 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.158674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.0.97.65:38907)
0927 18:32:10.158958 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.158968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.159294 (+ 326us) server_negotiation.cc:407] Connection header received
0927 18:32:10.159535 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.159540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.159610 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.159726 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.161186 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.162083 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.163495 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.163940 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.164082 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.168883 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.168889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.168893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.168932 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.171618 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.172258 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.172265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.172269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.172405 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.172817 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.172823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.172827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.173075 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.173330 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.173607 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.173864 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:10.218022 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.204337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.0.97.65:38907)
0927 18:32:10.204599 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.204605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.204819 (+ 214us) server_negotiation.cc:407] Connection header received
0927 18:32:10.204963 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.204967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.205031 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.205117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.206928 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.207603 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.208794 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.209226 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.209316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.213026 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.213030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.213034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.213072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.215617 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.216155 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.216159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.216161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.216288 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.216655 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.216658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.216660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.216952 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.217128 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.217430 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.217591 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 18:32:10.219875 1068 tablet_service.cc:2908] Rejecting scan request for tablet 70fdd2db24594e16ba6df2dbcd318e25: Uninitialized: clean time has not yet been initialized
I20260927 18:32:10.223376 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1025
I20260927 18:32:10.236898 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.224399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.0.97.65:38907)
0927 18:32:10.224574 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.224581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.224891 (+ 310us) server_negotiation.cc:407] Connection header received
0927 18:32:10.225089 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.225094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.225173 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.225274 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.226563 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.227237 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.228429 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.228866 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.229008 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.231266 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.231272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.231277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.231316 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.234234 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.234877 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.234885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.234888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.235093 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.235498 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.235505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.235508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.235821 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.236097 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.236375 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.236580 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:32:10.237488 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:10.325659 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1025
I20260927 18:32:10.343051 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1159
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:10.359090 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.343989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.0.97.66:38409)
0927 18:32:10.344158 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.344163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.344187 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:32:10.344242 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.344246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.344302 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.344385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.345709 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.346567 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.347929 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.348335 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.348431 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.352853 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.352867 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.352883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.352930 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.356144 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.356830 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.356839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.356841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.357010 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.357515 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.357519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.357521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.358135 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.358385 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.358663 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.358844 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:32:10.359830 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:10.450807 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1159
I20260927 18:32:10.467479 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1292
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:10.482380 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:10.468221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.0.97.67:42337)
0927 18:32:10.468388 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:10.468393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:10.468411 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:32:10.468537 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:10.468541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:10.468615 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:10.468718 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:10.470082 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.470831 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.472180 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:10.472635 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:10.472739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:10.476977 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:10.476993 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:10.477009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:10.477047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:10.479883 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.480535 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.480543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.480545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.480691 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:10.481032 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:10.481035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:10.481037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:10.481386 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:10.481587 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:10.481924 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:32:10.482093 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:32:10.483065 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:10.570444 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1292
I20260927 18:32:10.586766 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 932
I20260927 18:32:10.588029 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:10.700033 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 932
2026-09-27T18:32:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7589 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[1468](info): set up 2 sockets
Sep 27 18:32:10 dist-test-slave-n3p4 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260927 18:32:11.802016 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:11 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:11.987694 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40537
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:12.305939 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:12.306273 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:12.306447 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:12.313516 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:12.313661 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:32:12.313755 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:12.313894 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:12.313989 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:12.314078 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:12.323854 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40537
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:12.325132 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:12.326463 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:12.338800 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:12.338909 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:12.338943 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:12.339951 1481 server_base.cc:1034] running on GCE node
I20260927 18:32:12.340773 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:12.342429 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:12.343688 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790533932343616 us; error 51 us; skew 500 ppm
I20260927 18:32:12.344058 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:12 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:12.347275 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:12.348834 1481 webserver.cc:460] Webserver started at http://127.0.97.126:34605/ using document root <none> and password file <none>
I20260927 18:32:12.349535 1481 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:12.349666 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:12.350044 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:12.352922 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "fb06577fd845429cba2f6492198e5291"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "c3165f8b471c09a3a0c979c6f5a93055"
server_key_iv: "2ee6569b88bec6389d871192fe6a3731"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.353860 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "fb06577fd845429cba2f6492198e5291"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "c3165f8b471c09a3a0c979c6f5a93055"
server_key_iv: "2ee6569b88bec6389d871192fe6a3731"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.359042 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:12.362460 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:12.363832 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:12.364001 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "fb06577fd845429cba2f6492198e5291"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "c3165f8b471c09a3a0c979c6f5a93055"
server_key_iv: "2ee6569b88bec6389d871192fe6a3731"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.364215 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:12.387120 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:12.391588 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:12.391929 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:12.413668 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40537
I20260927 18:32:12.413687 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40537 every 8 connection(s)
I20260927 18:32:12.415114 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:12.418861 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:12.425231 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1481
I20260927 18:32:12.425472 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:12.426306 389 external_mini_cluster.cc:1428] Setting key e93c75a16d3623898ae353ecdf831a7f
I20260927 18:32:12.430095 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Bootstrap starting.
I20260927 18:32:12.433534 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:12.434700 1558 log.cc:824] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:12.437969 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: No bootstrap required, opened a new log
Sep 27 18:32:12 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:12.448565 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } }
I20260927 18:32:12.449025 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:12.449138 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb06577fd845429cba2f6492198e5291, State: Initialized, Role: FOLLOWER
I20260927 18:32:12.449859 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } }
I20260927 18:32:12.450156 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:12.450371 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:12.450587 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:12.452194 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:12.428577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.0.97.126:40537)
0927 18:32:12.429457 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:12.429470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:12.429515 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:32:12.431481 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:12.431510 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:12.431926 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:12.432272 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:12.434129 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.435849 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.437655 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.438509 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.438670 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:12.443316 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:12.443328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:12.443347 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:12.443395 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:12.447477 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:12.448202 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:12.448209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:12.448218 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:12.448409 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:12.448797 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:12.448802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:12.448805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:12.449505 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:12.449772 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:12.450277 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 18:32:12.450746 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":421,"threads_started":1}
I20260927 18:32:12.453419 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } }
I20260927 18:32:12.453981 1558 leader_election.cc:302] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb06577fd845429cba2f6492198e5291; no voters:
I20260927 18:32:12.454639 1558 leader_election.cc:288] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:12.454783 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:12.456135 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [term 1 LEADER]: Becoming Leader. State: Replica: fb06577fd845429cba2f6492198e5291, State: Running, Role: LEADER
I20260927 18:32:12.456656 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } }
I20260927 18:32:12.456802 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:12.459976 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb06577fd845429cba2f6492198e5291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } } }
I20260927 18:32:12.459967 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb06577fd845429cba2f6492198e5291. Latest consensus state: current_term: 1 leader_uuid: "fb06577fd845429cba2f6492198e5291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb06577fd845429cba2f6492198e5291" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40537 } } }
I20260927 18:32:12.460379 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:12.460377 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:12.462069 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:12.467319 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:12.476660 1570 catalog_manager.cc:1347] Generated new cluster ID: c990a558646d402087b9bce155af82f4
I20260927 18:32:12.476778 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:12.516868 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:12.518733 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:12.526253 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Generated new TSK 0
I20260927 18:32:12.527273 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:12.570025 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:12.902541 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:12.902899 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:12.903070 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:12.910691 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:12.910844 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:12.910988 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:12.911119 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:12.921065 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:40537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:12.922510 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:12.923851 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:12.937100 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:12.937041 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:12.937077 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:12.938304 1584 server_base.cc:1034] running on GCE node
I20260927 18:32:12.938902 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:12.939805 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:12.941038 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790533932940961 us; error 56 us; skew 500 ppm
I20260927 18:32:12.941345 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:12 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:12.944741 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:12.946329 1584 webserver.cc:460] Webserver started at http://127.0.97.65:35199/ using document root <none> and password file <none>
I20260927 18:32:12.947055 1584 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:12.947144 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:12.947463 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:12.950484 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "cc9336440ad04ede80187d1682fd6d1d"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "1e4301854add19ee33af4e76aa48abda"
server_key_iv: "dab7ccacff7756781763854267d6f422"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.951278 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9336440ad04ede80187d1682fd6d1d"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "1e4301854add19ee33af4e76aa48abda"
server_key_iv: "dab7ccacff7756781763854267d6f422"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.956682 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:12.960419 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:12.962006 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:32:12.962184 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "cc9336440ad04ede80187d1682fd6d1d"
format_stamp: "Formatted at 2026-09-27 18:32:12 on dist-test-slave-n3p4"
server_key: "1e4301854add19ee33af4e76aa48abda"
server_key_iv: "dab7ccacff7756781763854267d6f422"
server_key_version: "encryptionkey@0"
I20260927 18:32:12.962440 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:12.978364 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:12.982961 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:12.983253 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:12.985829 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:12.985940 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:12.986078 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:12.986163 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:32:13 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:13.017390 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:12.986637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58969 (local address 127.0.97.126:40537)
0927 18:32:12.987233 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:12.987239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:12.988451 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 18:32:12.990887 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:12.990893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:12.990987 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:12.991110 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:12.994037 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.995023 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.998433 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.998899 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.999048 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.004247 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:13.004252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:13.004255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:13.004289 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:13.013362 (+ 9073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.014404 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.014412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.014419 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.014585 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.015147 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.015154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.015158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.015592 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:13.015774 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:13.016796 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0927 18:32:13.017017 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":227,"threads_started":1}
I20260927 18:32:13.019109 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:12.987286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.65:58969)
0927 18:32:12.988246 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:12.988295 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:12.990417 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:12.991457 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:12.991474 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:12.992616 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:12.993801 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:12.993841 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.995215 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.995224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:12.998241 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:12.998262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:12.999160 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:12.999169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:12.999539 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.000534 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:13.000583 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:13.003973 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:13.013518 (+ 9545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.013528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.013547 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.014261 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.014749 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.014753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.014755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.014946 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.015760 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:13.015774 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:13.016313 (+ 539us) client_negotiation.cc:769] Sending connection context
0927 18:32:13.016709 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 18:32:13.017296 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":418,"threads_started":1}
I20260927 18:32:13.027110 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44037
I20260927 18:32:13.027240 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44037 every 8 connection(s)
I20260927 18:32:13.028539 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:13.030957 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40537
I20260927 18:32:13.031189 1720 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:13.031977 1720 heartbeater.cc:511] Master 127.0.97.126:40537 requested a full tablet report, sending...
I20260927 18:32:13.034323 1513 ts_manager.cc:194] Registered new tserver with Master: cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037)
I20260927 18:32:13.036412 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:58969
I20260927 18:32:13.038328 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1584
I20260927 18:32:13.038578 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:13.039389 389 external_mini_cluster.cc:1428] Setting key 34692baf60f733c41985645c806281f0
I20260927 18:32:13.040691 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.026532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.65:52203)
0927 18:32:13.027587 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:13.027602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:13.027763 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:13.028103 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:13.028107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:13.028565 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:13.028920 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.028930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.030232 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.030238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.031718 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.031731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.032495 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.032506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.032699 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.033356 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:13.033378 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:13.034417 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:13.037899 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.037907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.037914 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.038530 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.038942 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.038947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.038951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.039139 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.039764 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:13.039769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:13.039948 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 18:32:13.040122 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:32:13.040345 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":557,"threads_started":1}
I20260927 18:32:13.040804 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.027027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52203 (local address 127.0.97.126:40537)
0927 18:32:13.027247 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:13.027254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:13.027692 (+ 438us) server_negotiation.cc:407] Connection header received
0927 18:32:13.027906 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:13.027911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:13.027977 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:13.028082 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:13.029058 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.030079 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.031854 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.032319 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.032417 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.034610 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:13.034616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:13.034620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:13.034663 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:13.037690 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.038668 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.038673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.038676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.038815 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.039265 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.039269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.039273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.039598 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:13.039771 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:13.040233 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 18:32:13.040476 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:13.081350 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:13.408499 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:13.408783 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:13.408879 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:13.416366 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:13.416529 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:13.416687 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:13.416826 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:13.426992 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:40537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:13.428277 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:13.429641 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:13.443032 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:13.443011 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:13.443439 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:13.443607 1724 server_base.cc:1034] running on GCE node
I20260927 18:32:13.444371 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:13.445326 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:13.446570 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790533933446481 us; error 52 us; skew 500 ppm
I20260927 18:32:13.446943 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:13 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:13.450609 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:13.452180 1724 webserver.cc:460] Webserver started at http://127.0.97.66:33171/ using document root <none> and password file <none>
I20260927 18:32:13.452864 1724 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:13.452957 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:13.453244 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:13.456329 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "487898b24c524b408fb7206fb52154fc"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "9074be2b3a7569fd37e8543ecd1bca07"
server_key_iv: "7a8960bec38542b29539fc25e31391f5"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.457216 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "487898b24c524b408fb7206fb52154fc"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "9074be2b3a7569fd37e8543ecd1bca07"
server_key_iv: "7a8960bec38542b29539fc25e31391f5"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.462890 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 18:32:13.466567 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:13.468175 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:13.468300 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "487898b24c524b408fb7206fb52154fc"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "9074be2b3a7569fd37e8543ecd1bca07"
server_key_iv: "7a8960bec38542b29539fc25e31391f5"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.468474 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:13.487442 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:13.491863 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:13.492095 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:13.494833 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:13.494915 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:13.495079 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:13.495139 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:32:13 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:13.519809 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.495438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:45247 (local address 127.0.97.126:40537)
0927 18:32:13.495675 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:13.495681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:13.497223 (+ 1542us) server_negotiation.cc:407] Connection header received
0927 18:32:13.499744 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:13.499751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:13.499827 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:13.499885 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:13.502451 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.503796 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.506217 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.507033 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.507204 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.512100 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:13.512111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:13.512117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:13.512165 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:13.515668 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.516922 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.516928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.516932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.517120 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.517656 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.517662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.517665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.518022 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:13.518214 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:13.519231 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0927 18:32:13.519451 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 18:32:13.521268 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.496080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.66:45247)
0927 18:32:13.497024 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:13.497079 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:13.499341 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:13.500464 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:13.500472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:13.501330 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:13.502177 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.502201 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.504024 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.504031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.505997 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.506013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.507310 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.507320 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.507638 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.508580 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:13.508602 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:13.511779 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:13.515845 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.515860 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.515887 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.516728 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.517264 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.517270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.517275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.517510 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.518306 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:13.518316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:13.518821 (+ 505us) client_negotiation.cc:769] Sending connection context
0927 18:32:13.519137 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 18:32:13.519710 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":417,"threads_started":1}
I20260927 18:32:13.543062 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38881
I20260927 18:32:13.543038 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38881 every 8 connection(s)
I20260927 18:32:13.543489 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.527041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:54879 (local address 127.0.97.126:40537)
0927 18:32:13.527254 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:13.527263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:13.528021 (+ 758us) server_negotiation.cc:407] Connection header received
0927 18:32:13.528211 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:13.528217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:13.528339 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:13.528440 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:13.529987 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.530797 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.533371 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.533877 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.533971 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.536787 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:13.536792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:13.536795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:13.536824 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:13.540500 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.541338 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.541342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.541345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.541463 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:13.541911 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:13.541916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:13.541919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:13.542313 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:13.542462 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:13.542924 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 18:32:13.543149 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:13.543651 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:13.527098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.66:54879)
0927 18:32:13.527878 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:13.527903 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:13.528062 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:13.528492 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:13.528497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:13.529269 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:13.529818 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.529833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.530962 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.530973 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.533236 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:13.533254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:13.534068 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:13.534075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:13.534280 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:13.535039 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:13.535070 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:13.536591 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:13.540668 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.540676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.540683 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.541224 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.541584 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:13.541589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:13.541593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:13.541770 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:13.542454 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:13.542459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:13.542662 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:32:13.542827 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 18:32:13.543168 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":416,"threads_started":1}
I20260927 18:32:13.544716 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:13.546845 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40537
I20260927 18:32:13.547081 1859 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:13.547775 1859 heartbeater.cc:511] Master 127.0.97.126:40537 requested a full tablet report, sending...
I20260927 18:32:13.549032 1513 ts_manager.cc:194] Registered new tserver with Master: 487898b24c524b408fb7206fb52154fc (127.0.97.66:38881)
I20260927 18:32:13.549445 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1724
I20260927 18:32:13.549616 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:13.550019 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:45247
I20260927 18:32:13.550113 389 external_mini_cluster.cc:1428] Setting key ba5e9401105f43d71dc27e14e731e02d
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:13.589020 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:13.914250 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:13.914531 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:13.914700 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:13.921975 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:13.922096 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:13.922276 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:13.922405 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:13.932276 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:40537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:13.933694 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:13.935060 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:13.948616 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:13.948539 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:13.948663 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:13.949342 1863 server_base.cc:1034] running on GCE node
I20260927 18:32:13.950050 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:13.951014 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:13.952251 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790533933952190 us; error 37 us; skew 500 ppm
I20260927 18:32:13.952595 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:13 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:13.955914 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:13.957402 1863 webserver.cc:460] Webserver started at http://127.0.97.67:34937/ using document root <none> and password file <none>
I20260927 18:32:13.958156 1863 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:13.958246 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:13.958506 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:13.961328 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "7b659659bb2c400288947436e49064fe"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "6baddb8e92c989631fbea585472b6318"
server_key_iv: "ef11e1a212038aab2719d601ff55ba72"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.962157 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "7b659659bb2c400288947436e49064fe"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "6baddb8e92c989631fbea585472b6318"
server_key_iv: "ef11e1a212038aab2719d601ff55ba72"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.967607 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:13.971197 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:13.972651 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:13.972824 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "7b659659bb2c400288947436e49064fe"
format_stamp: "Formatted at 2026-09-27 18:32:13 on dist-test-slave-n3p4"
server_key: "6baddb8e92c989631fbea585472b6318"
server_key_iv: "ef11e1a212038aab2719d601ff55ba72"
server_key_version: "encryptionkey@0"
I20260927 18:32:13.973125 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:14.003723 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:14.008656 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:14.008962 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:14.011855 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:14.011967 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:14.012084 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:14.012197 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 18:32:14 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:14.039698 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.012654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38035 (local address 127.0.97.126:40537)
0927 18:32:14.012868 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.012878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.014422 (+ 1544us) server_negotiation.cc:407] Connection header received
0927 18:32:14.016870 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.016876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.016978 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.017102 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:14.019819 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.020943 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.024466 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.024933 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.025049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.030757 (+ 5708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:14.030765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:14.030770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:14.030814 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:14.035370 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.036307 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.036316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.036351 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.036541 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.037201 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.037208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.037212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.037756 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:14.037998 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.039005 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.039284 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:32:14.039954 1720 heartbeater.cc:503] Master 127.0.97.126:40537 was elected leader, sending a full tablet report...
I20260927 18:32:14.041121 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.013313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.67:38035)
0927 18:32:14.014248 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.014293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.016506 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.017473 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.017485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.018498 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:14.019568 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.019604 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.021221 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.021234 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.024295 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.024315 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.025214 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.025225 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.025585 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.026610 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:14.026654 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:14.030450 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:14.035624 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.035634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.035649 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.036152 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.036693 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.036701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.036707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.037046 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.038096 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:14.038107 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:14.038546 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.038859 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.039481 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":441,"threads_started":1}
I20260927 18:32:14.067756 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.049999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.97.67:56597)
0927 18:32:14.050952 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.050972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.051171 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.051520 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.051524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.052305 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:14.052986 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.052998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.054281 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.054286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.056706 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.056727 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.057590 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.057600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.057885 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.058760 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:14.058801 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:14.059988 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:14.064271 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.064276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.064280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.064919 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.065428 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.065433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.065436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.065671 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.066441 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:14.066444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:14.066665 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.066868 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.067293 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":623,"threads_started":1}
I20260927 18:32:14.068464 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.049562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56597 (local address 127.0.97.126:40537)
0927 18:32:14.049831 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.049839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.051082 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 18:32:14.051312 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.051317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.051388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.051489 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:14.053149 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.054081 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.056880 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.057336 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.057494 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.060213 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:14.060223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:14.060231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:14.060272 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:14.064124 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.065142 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.065151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.065156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.065326 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.065810 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.065815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.065822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.066215 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:14.066379 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.067874 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.068146 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:32:14.070896 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37957
I20260927 18:32:14.072264 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:14.070900 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37957 every 8 connection(s)
I20260927 18:32:14.074572 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40537
I20260927 18:32:14.074767 1998 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:14.075336 1998 heartbeater.cc:511] Master 127.0.97.126:40537 requested a full tablet report, sending...
I20260927 18:32:14.076619 1513 ts_manager.cc:194] Registered new tserver with Master: 7b659659bb2c400288947436e49064fe (127.0.97.67:37957)
I20260927 18:32:14.077440 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 1863
I20260927 18:32:14.077485 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:38035
I20260927 18:32:14.077688 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:14.078166 389 external_mini_cluster.cc:1428] Setting key 4187f1a4b8e3a34935948faf6d014932
I20260927 18:32:14.081126 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:32:14 dist-test-slave-n3p4 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:32:14 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:14.126897 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.112597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.126:40537)
0927 18:32:14.112768 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.112772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.113008 (+ 236us) server_negotiation.cc:407] Connection header received
0927 18:32:14.113217 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.113222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.113295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.113404 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:14.114855 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.115654 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.116895 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.117350 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.117460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.121175 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:14.121180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:14.121183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:14.121218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:14.124420 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.125026 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.125031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.125035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.125165 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.125486 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.125489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.125491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.125881 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:14.126095 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.126409 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.126616 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:32:14.131235 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:14.138794 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:14.176754 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.152191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.0.97.67:37957)
0927 18:32:14.167626 (+ 15435us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.168354 (+ 728us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.168404 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:32:14.168503 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.168511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.168714 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.168899 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.169999 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171266 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.174023 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.175497 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.175631 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.175885 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.176260 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.176419 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15248,"thread_start_us":15438,"threads_started":1}
I20260927 18:32:14.176827 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.151837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37957 (local address 127.0.0.1:49320)
0927 18:32:14.165236 (+ 13399us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.165285 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.165489 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.168940 (+ 3451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.168950 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.169042 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.169715 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.169734 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171389 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.171397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.173881 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.173892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.175717 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.175725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.175987 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.176041 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.176165 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.176398 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13089,"mutex_wait_us":150,"spinlock_wait_cycles":73600,"thread_start_us":15358,"threads_started":1}
I20260927 18:32:14.178866 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.152444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.0.97.66:38881)
0927 18:32:14.167136 (+ 14692us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.167146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.167622 (+ 476us) server_negotiation.cc:407] Connection header received
0927 18:32:14.168183 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.168195 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.168473 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.168645 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.169817 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171346 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.174397 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.175749 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.175876 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.176051 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.178426 (+ 2375us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.178586 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14513,"thread_start_us":14372,"threads_started":1}
I20260927 18:32:14.178987 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.152078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38881 (local address 127.0.0.1:38768)
0927 18:32:14.167507 (+ 15429us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.167525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.167953 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.168941 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.168949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.168999 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.169542 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.169559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171539 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.171550 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.174237 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.174251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.178014 (+ 3763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.178021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.178212 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.178256 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.178356 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.178525 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15274,"mutex_wait_us":146,"thread_start_us":13660,"threads_started":1}
I20260927 18:32:14.179965 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.167602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44037 (local address 127.0.0.1:34182)
0927 18:32:14.168214 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.168231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.168339 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.169195 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.169199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.169233 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.169943 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.169954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171642 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.171648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.174246 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.174254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.177710 (+ 3456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.177717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.179342 (+ 1625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.179375 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.179473 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.179594 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":263,"thread_start_us":305,"threads_started":1}
I20260927 18:32:14.179957 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.167841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.0.97.65:44037)
0927 18:32:14.168431 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.168439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.168464 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:32:14.168552 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.168560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.168781 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.169014 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.170112 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.171507 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.176150 (+ 4643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.177514 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.177670 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.177892 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.179531 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.179693 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
I20260927 18:32:14.181051 1927 tablet_service.cc:1467] Processing CreateTablet for tablet f6accf11ee78442fb42df4508ca78428 (DEFAULT_TABLE table=test-table [id=a4402511f6a34910bbb162cb5f63ed3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.182374 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6accf11ee78442fb42df4508ca78428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.182524 1789 tablet_service.cc:1467] Processing CreateTablet for tablet f6accf11ee78442fb42df4508ca78428 (DEFAULT_TABLE table=test-table [id=a4402511f6a34910bbb162cb5f63ed3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.183849 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6accf11ee78442fb42df4508ca78428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.183668 1650 tablet_service.cc:1467] Processing CreateTablet for tablet f6accf11ee78442fb42df4508ca78428 (DEFAULT_TABLE table=test-table [id=a4402511f6a34910bbb162cb5f63ed3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.184909 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6accf11ee78442fb42df4508ca78428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.193101 2017 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Bootstrap starting.
I20260927 18:32:14.196192 2017 tablet_bootstrap.cc:654] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.197252 2017 log.cc:824] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:14.198190 2018 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Bootstrap starting.
I20260927 18:32:14.201725 2018 tablet_bootstrap.cc:654] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.201822 2017 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: No bootstrap required, opened a new log
I20260927 18:32:14.202081 2017 ts_tablet_manager.cc:1397] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 18:32:14.202994 2018 log.cc:824] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:14.204208 2020 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Bootstrap starting.
I20260927 18:32:14.205929 2018 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: No bootstrap required, opened a new log
I20260927 18:32:14.206184 2018 ts_tablet_manager.cc:1397] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260927 18:32:14.207724 2020 tablet_bootstrap.cc:654] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.208966 2020 log.cc:824] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:14.212751 2020 tablet_bootstrap.cc:492] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: No bootstrap required, opened a new log
I20260927 18:32:14.213030 2020 ts_tablet_manager.cc:1397] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 18:32:14.212949 2017 raft_consensus.cc:348] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.213368 2017 raft_consensus.cc:374] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.213469 2017 raft_consensus.cc:729] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b659659bb2c400288947436e49064fe, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.214118 2017 consensus_queue.cc:260] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.215204 1998 heartbeater.cc:503] Master 127.0.97.126:40537 was elected leader, sending a full tablet report...
I20260927 18:32:14.215917 2017 ts_tablet_manager.cc:1428] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 18:32:14.215943 2018 raft_consensus.cc:348] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.216354 2018 raft_consensus.cc:374] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.216451 2018 raft_consensus.cc:729] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487898b24c524b408fb7206fb52154fc, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.217053 2018 consensus_queue.cc:260] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.218128 1859 heartbeater.cc:503] Master 127.0.97.126:40537 was elected leader, sending a full tablet report...
I20260927 18:32:14.220028 2018 ts_tablet_manager.cc:1428] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260927 18:32:14.223742 2020 raft_consensus.cc:348] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.224274 2020 raft_consensus.cc:374] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.224385 2020 raft_consensus.cc:729] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9336440ad04ede80187d1682fd6d1d, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.224893 2020 consensus_queue.cc:260] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.226549 2020 ts_tablet_manager.cc:1428] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
W20260927 18:32:14.281121 1721 tablet.cc:2369] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:32:14.297025 1860 tablet.cc:2369] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:14.324118 2024 raft_consensus.cc:482] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:14.324427 2024 raft_consensus.cc:504] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
W20260927 18:32:14.324753 1999 tablet.cc:2369] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:14.326139 2024 leader_election.cc:288] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b659659bb2c400288947436e49064fe (127.0.97.67:37957), cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037)
I20260927 18:32:14.333559 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.326642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:59259 (local address 127.0.97.67:37957)
0927 18:32:14.326817 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.326821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.327290 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:32:14.327664 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.327671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.327756 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.327828 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.328543 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.329273 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.331751 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.332520 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.332717 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.332832 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.333171 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.333301 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:32:14.333555 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.327497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35255 (local address 127.0.97.65:44037)
0927 18:32:14.327755 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.327761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.327974 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:32:14.328236 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.328241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.328311 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.328407 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.328924 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.329600 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.331807 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.332578 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.332734 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.332855 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.333171 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.333300 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 18:32:14.333698 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.327278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44037 (local address 127.0.97.66:35255)
0927 18:32:14.327864 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.327877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.328078 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.328419 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.328423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.328445 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.328767 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.328774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.329736 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.329743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.331652 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.331663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.332803 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.332808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.333000 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.333037 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.333147 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.333294 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":5,"spinlock_wait_cycles":29056,"thread_start_us":294,"threads_started":1}
I20260927 18:32:14.334167 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.326482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37957 (local address 127.0.97.66:59259)
0927 18:32:14.327185 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.327206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.327424 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.328000 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.328003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.328037 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.328390 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.328399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.329426 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.329432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.331547 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.331596 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.332784 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.332790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.332997 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.333040 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.333146 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.333304 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":78,"thread_start_us":351,"threads_started":1}
I20260927 18:32:14.334792 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6accf11ee78442fb42df4508ca78428" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9336440ad04ede80187d1682fd6d1d" is_pre_election: true
I20260927 18:32:14.335312 1670 raft_consensus.cc:2454] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 0.
I20260927 18:32:14.335342 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6accf11ee78442fb42df4508ca78428" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe" is_pre_election: true
I20260927 18:32:14.335897 1953 raft_consensus.cc:2454] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 0.
I20260927 18:32:14.335983 1743 leader_election.cc:302] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487898b24c524b408fb7206fb52154fc, cc9336440ad04ede80187d1682fd6d1d; no voters:
I20260927 18:32:14.336431 2024 raft_consensus.cc:2781] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:14.336660 2024 raft_consensus.cc:482] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:14.336783 2024 raft_consensus.cc:3037] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.339016 2024 raft_consensus.cc:504] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.339623 2024 leader_election.cc:288] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 election: Requested vote from peers 7b659659bb2c400288947436e49064fe (127.0.97.67:37957), cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037)
I20260927 18:32:14.340106 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6accf11ee78442fb42df4508ca78428" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9336440ad04ede80187d1682fd6d1d"
I20260927 18:32:14.340175 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6accf11ee78442fb42df4508ca78428" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe"
I20260927 18:32:14.340327 1670 raft_consensus.cc:3037] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.340353 1953 raft_consensus.cc:3037] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.343930 1670 raft_consensus.cc:2454] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 1.
I20260927 18:32:14.343961 1953 raft_consensus.cc:2454] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 1.
I20260927 18:32:14.344368 1743 leader_election.cc:302] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487898b24c524b408fb7206fb52154fc, cc9336440ad04ede80187d1682fd6d1d; no voters:
I20260927 18:32:14.344728 2024 raft_consensus.cc:2781] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:14.345237 2024 raft_consensus.cc:686] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 1 LEADER]: Becoming Leader. State: Replica: 487898b24c524b408fb7206fb52154fc, State: Running, Role: LEADER
I20260927 18:32:14.345815 2024 consensus_queue.cc:237] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } }
I20260927 18:32:14.350785 1512 catalog_manager.cc:5495] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 487898b24c524b408fb7206fb52154fc (127.0.97.66). New cstate: current_term: 1 leader_uuid: "487898b24c524b408fb7206fb52154fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:14.415334 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.409923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.0.97.66:38881)
0927 18:32:14.410161 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.410169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.410195 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:32:14.410448 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.410452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.410530 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.410662 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:14.410991 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.411794 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.413120 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.413528 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.413628 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.414266 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.414792 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.414896 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.415017 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.415081 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:32:14.424218 1953 raft_consensus.cc:1261] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Refusing update from remote peer 487898b24c524b408fb7206fb52154fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.424223 1670 raft_consensus.cc:1261] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Refusing update from remote peer 487898b24c524b408fb7206fb52154fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.425148 2028 consensus_queue.cc:1035] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.425532 2024 consensus_queue.cc:1035] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.436962 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334026995390623744 to 7334026995084357632. Current Snapshot: MvccSnapshot[applied={T|T < 7334026995390623744}]
I20260927 18:32:14.445480 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334026995390623744 to 7334026995084357632. Current Snapshot: MvccSnapshot[applied={T|T < 7334026995390623744}]
I20260927 18:32:14.459182 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:14.459479 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:14.463534 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Sending DeleteTablet for 3 replicas of tablet f6accf11ee78442fb42df4508ca78428
I20260927 18:32:14.464643 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet f6accf11ee78442fb42df4508ca78428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38768
I20260927 18:32:14.464676 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet f6accf11ee78442fb42df4508ca78428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49320
I20260927 18:32:14.465370 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet f6accf11ee78442fb42df4508ca78428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34182
I20260927 18:32:14.465637 2040 tablet_replica.cc:330] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: stopping tablet replica
I20260927 18:32:14.466260 2040 raft_consensus.cc:2229] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:14.466889 2040 raft_consensus.cc:2258] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.467857 2041 tablet_replica.cc:330] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: stopping tablet replica
I20260927 18:32:14.468384 2041 raft_consensus.cc:2229] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:14.468480 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:14.469097 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:14.469251 2040 ts_tablet_manager.cc:1905] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.469401 2041 raft_consensus.cc:2258] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.471558 2041 ts_tablet_manager.cc:1905] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.476008 2040 ts_tablet_manager.cc:1918] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:14.477149 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e (DEFAULT_TABLE table=test-table [id=c64bd0e946a2415bafb609be44ee0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.477341 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e (DEFAULT_TABLE table=test-table [id=c64bd0e946a2415bafb609be44ee0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.477597 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170966a3e3394e9fbf38d396f7d33a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.477877 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170966a3e3394e9fbf38d396f7d33a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.482537 2018 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Bootstrap starting.
I20260927 18:32:14.484051 2020 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Bootstrap starting.
I20260927 18:32:14.484848 2018 tablet_bootstrap.cc:654] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.485642 2020 tablet_bootstrap.cc:654] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.486516 2042 tablet_replica.cc:330] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: stopping tablet replica
I20260927 18:32:14.487038 2042 raft_consensus.cc:2229] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:14.487638 2042 raft_consensus.cc:2258] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.482671 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e (DEFAULT_TABLE table=test-table [id=c64bd0e946a2415bafb609be44ee0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:14.488147 2040 log.cc:1196] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/f6accf11ee78442fb42df4508ca78428
I20260927 18:32:14.488444 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170966a3e3394e9fbf38d396f7d33a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.490590 2041 ts_tablet_manager.cc:1918] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:14.491731 2042 ts_tablet_manager.cc:1905] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.492810 2041 log.cc:1196] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/f6accf11ee78442fb42df4508ca78428
I20260927 18:32:14.493530 2041 ts_tablet_manager.cc:1939] T f6accf11ee78442fb42df4508ca78428 P 487898b24c524b408fb7206fb52154fc: Deleting consensus metadata
I20260927 18:32:14.494002 2040 ts_tablet_manager.cc:1939] T f6accf11ee78442fb42df4508ca78428 P 7b659659bb2c400288947436e49064fe: Deleting consensus metadata
I20260927 18:32:14.495133 2020 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: No bootstrap required, opened a new log
I20260927 18:32:14.495271 2020 ts_tablet_manager.cc:1397] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20260927 18:32:14.495805 1500 catalog_manager.cc:4841] TS 7b659659bb2c400288947436e49064fe (127.0.97.67:37957): tablet f6accf11ee78442fb42df4508ca78428 (table test-table [id=a4402511f6a34910bbb162cb5f63ed3f]) successfully deleted
I20260927 18:32:14.496282 2017 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Bootstrap starting.
I20260927 18:32:14.496524 2020 raft_consensus.cc:348] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.496755 2020 raft_consensus.cc:374] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.496882 2020 raft_consensus.cc:729] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9336440ad04ede80187d1682fd6d1d, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.496912 1499 catalog_manager.cc:4841] TS 487898b24c524b408fb7206fb52154fc (127.0.97.66:38881): tablet f6accf11ee78442fb42df4508ca78428 (table test-table [id=a4402511f6a34910bbb162cb5f63ed3f]) successfully deleted
I20260927 18:32:14.497640 2020 consensus_queue.cc:260] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.498332 2017 tablet_bootstrap.cc:654] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.498982 2020 ts_tablet_manager.cc:1428] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:32:14.503067 2018 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: No bootstrap required, opened a new log
I20260927 18:32:14.503283 2018 ts_tablet_manager.cc:1397] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.003s
I20260927 18:32:14.503450 2042 ts_tablet_manager.cc:1918] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:14.503634 2042 log.cc:1196] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/f6accf11ee78442fb42df4508ca78428
I20260927 18:32:14.504678 2042 ts_tablet_manager.cc:1939] T f6accf11ee78442fb42df4508ca78428 P cc9336440ad04ede80187d1682fd6d1d: Deleting consensus metadata
I20260927 18:32:14.504848 2018 raft_consensus.cc:348] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.505085 2018 raft_consensus.cc:374] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.505193 2018 raft_consensus.cc:729] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487898b24c524b408fb7206fb52154fc, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.505474 2018 consensus_queue.cc:260] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.506111 2018 ts_tablet_manager.cc:1428] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 18:32:14.506929 1500 catalog_manager.cc:4841] TS cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037): tablet f6accf11ee78442fb42df4508ca78428 (table test-table [id=a4402511f6a34910bbb162cb5f63ed3f]) successfully deleted
I20260927 18:32:14.507730 2017 tablet_bootstrap.cc:492] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: No bootstrap required, opened a new log
I20260927 18:32:14.507876 2017 ts_tablet_manager.cc:1397] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 18:32:14.509093 2017 raft_consensus.cc:348] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.509328 2017 raft_consensus.cc:374] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.509419 2017 raft_consensus.cc:729] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b659659bb2c400288947436e49064fe, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.509709 2017 consensus_queue.cc:260] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.510308 2017 ts_tablet_manager.cc:1428] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:32:14.527637 2024 raft_consensus.cc:482] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:14.527884 2024 raft_consensus.cc:504] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.528677 2024 leader_election.cc:288] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037), 7b659659bb2c400288947436e49064fe (127.0.97.67:37957)
I20260927 18:32:14.529173 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170966a3e3394e9fbf38d396f7d33a3e" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9336440ad04ede80187d1682fd6d1d" is_pre_election: true
I20260927 18:32:14.529305 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170966a3e3394e9fbf38d396f7d33a3e" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe" is_pre_election: true
I20260927 18:32:14.529443 1670 raft_consensus.cc:2454] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 0.
I20260927 18:32:14.529623 1952 raft_consensus.cc:2454] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 0.
I20260927 18:32:14.529981 1743 leader_election.cc:302] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487898b24c524b408fb7206fb52154fc, cc9336440ad04ede80187d1682fd6d1d; no voters:
I20260927 18:32:14.530290 2024 raft_consensus.cc:2781] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:14.530416 2024 raft_consensus.cc:482] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:14.530514 2024 raft_consensus.cc:3037] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.533581 2024 raft_consensus.cc:504] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.534230 2024 leader_election.cc:288] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 election: Requested vote from peers cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037), 7b659659bb2c400288947436e49064fe (127.0.97.67:37957)
I20260927 18:32:14.534799 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170966a3e3394e9fbf38d396f7d33a3e" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe"
I20260927 18:32:14.534852 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170966a3e3394e9fbf38d396f7d33a3e" candidate_uuid: "487898b24c524b408fb7206fb52154fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9336440ad04ede80187d1682fd6d1d"
I20260927 18:32:14.534996 1952 raft_consensus.cc:3037] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.535046 1670 raft_consensus.cc:3037] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.537143 1952 raft_consensus.cc:2454] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 1.
I20260927 18:32:14.537544 1743 leader_election.cc:302] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487898b24c524b408fb7206fb52154fc, 7b659659bb2c400288947436e49064fe; no voters:
I20260927 18:32:14.537865 2024 raft_consensus.cc:2781] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:14.537937 1670 raft_consensus.cc:2454] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487898b24c524b408fb7206fb52154fc in term 1.
I20260927 18:32:14.538036 2024 raft_consensus.cc:686] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 1 LEADER]: Becoming Leader. State: Replica: 487898b24c524b408fb7206fb52154fc, State: Running, Role: LEADER
I20260927 18:32:14.538466 2024 consensus_queue.cc:237] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.541646 1512 catalog_manager.cc:5495] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 487898b24c524b408fb7206fb52154fc (127.0.97.66). New cstate: current_term: 1 leader_uuid: "487898b24c524b408fb7206fb52154fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:32:14 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:14.599501 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.584287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40537 (local address 127.0.0.1:49116)
0927 18:32:14.585420 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.585444 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.585572 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.585925 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.585928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.586579 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:14.586841 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.586850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.588313 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.588317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.589281 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.589291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.590286 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.590291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.590407 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.591097 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:14.591132 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:14.593215 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:14.596934 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.596943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.596948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.597511 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.597903 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:14.597906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:14.597908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:14.598031 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:14.598681 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:14.598689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:14.598837 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.598970 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.599177 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":849,"threads_started":1}
I20260927 18:32:14.599808 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.584360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.0.97.126:40537)
0927 18:32:14.584563 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.584570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.585543 (+ 973us) server_negotiation.cc:407] Connection header received
0927 18:32:14.585688 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.585694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.585771 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.585905 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:14.586990 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.588164 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.589427 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.590124 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.590261 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.593414 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:14.593421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:14.593426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:14.593462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:14.596808 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.597626 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.597632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.597636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.597797 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:14.598154 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:14.598159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:14.598163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:14.598510 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:14.598649 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.599103 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.599308 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:32:14.605685 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49116:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:32:14.606993 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:14.614115 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 902e17d758d24e34bd2096d37a1fd36f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4aa7319e84974a71b6daea08c7cf0556]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:14.614218 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 902e17d758d24e34bd2096d37a1fd36f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4aa7319e84974a71b6daea08c7cf0556]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:14.614655 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e17d758d24e34bd2096d37a1fd36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.614821 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e17d758d24e34bd2096d37a1fd36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.618148 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 902e17d758d24e34bd2096d37a1fd36f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4aa7319e84974a71b6daea08c7cf0556]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:14.618685 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e17d758d24e34bd2096d37a1fd36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:14.621910 2018 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: Bootstrap starting.
I20260927 18:32:14.622013 2020 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: Bootstrap starting.
I20260927 18:32:14.623624 2018 tablet_bootstrap.cc:654] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.623629 2020 tablet_bootstrap.cc:654] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.626693 2017 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: Bootstrap starting.
I20260927 18:32:14.628654 2017 tablet_bootstrap.cc:654] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:14.632709 2017 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: No bootstrap required, opened a new log
I20260927 18:32:14.632889 2017 ts_tablet_manager.cc:1397] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260927 18:32:14.633054 2018 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: No bootstrap required, opened a new log
I20260927 18:32:14.633224 2018 ts_tablet_manager.cc:1397] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260927 18:32:14.633562 2017 raft_consensus.cc:348] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.633771 2017 raft_consensus.cc:374] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.633877 2017 raft_consensus.cc:729] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b659659bb2c400288947436e49064fe, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.634111 2017 consensus_queue.cc:260] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.634603 2020 tablet_bootstrap.cc:492] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: No bootstrap required, opened a new log
I20260927 18:32:14.634487 2023 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.634755 2023 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.634779 2020 ts_tablet_manager.cc:1397] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260927 18:32:14.635562 2017 ts_tablet_manager.cc:1428] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:14.635964 2020 raft_consensus.cc:348] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.636241 2020 raft_consensus.cc:374] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.636314 2020 raft_consensus.cc:729] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9336440ad04ede80187d1682fd6d1d, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.636713 2018 raft_consensus.cc:348] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.636979 2018 raft_consensus.cc:374] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:14.637076 2018 raft_consensus.cc:729] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487898b24c524b408fb7206fb52154fc, State: Initialized, Role: FOLLOWER
I20260927 18:32:14.637459 2018 consensus_queue.cc:260] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.638219 2018 ts_tablet_manager.cc:1428] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:32:14.638054 2024 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.638451 2024 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.638912 2020 consensus_queue.cc:260] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.639664 2020 ts_tablet_manager.cc:1428] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:32:14.639545 2025 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.639953 2025 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.732209 2025 raft_consensus.cc:482] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:14.732482 2025 raft_consensus.cc:504] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.734009 2025 leader_election.cc:288] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 487898b24c524b408fb7206fb52154fc (127.0.97.66:38881), 7b659659bb2c400288947436e49064fe (127.0.97.67:37957)
I20260927 18:32:14.743388 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.734684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39021 (local address 127.0.97.67:37957)
0927 18:32:14.734866 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.734872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.736174 (+ 1302us) server_negotiation.cc:407] Connection header received
0927 18:32:14.736397 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.736401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.736471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.736571 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.737724 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.738491 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.741042 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.742314 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.742457 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.742556 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.742903 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.743043 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:32:14.743690 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.734633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41125 (local address 127.0.97.66:38881)
0927 18:32:14.735301 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.735309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.735930 (+ 621us) server_negotiation.cc:407] Connection header received
0927 18:32:14.736239 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.736245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.736330 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.736811 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:14.737667 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.738911 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.741178 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.742674 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.742824 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.742933 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.743257 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.743391 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510}
I20260927 18:32:14.743768 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.734256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38881 (local address 127.0.97.65:41125)
0927 18:32:14.735807 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.735830 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.736045 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.736842 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.736847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.736876 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.737370 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.737389 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.739042 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.739047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.741028 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.741043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.742898 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.742907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.743096 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.743130 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.743229 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.743370 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":43,"thread_start_us":1231,"threads_started":1}
I20260927 18:32:14.744573 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.734283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37957 (local address 127.0.97.65:39021)
0927 18:32:14.736087 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.736104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.736258 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.736968 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.736974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.737011 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:14.737496 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.737508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.738628 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.738634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.740863 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.740877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.742529 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.742536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.742699 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.742745 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.744033 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.744170 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":6,"thread_start_us":1534,"threads_started":1}
I20260927 18:32:14.745030 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e17d758d24e34bd2096d37a1fd36f" candidate_uuid: "cc9336440ad04ede80187d1682fd6d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487898b24c524b408fb7206fb52154fc" is_pre_election: true
I20260927 18:32:14.745291 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e17d758d24e34bd2096d37a1fd36f" candidate_uuid: "cc9336440ad04ede80187d1682fd6d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe" is_pre_election: true
I20260927 18:32:14.745579 1952 raft_consensus.cc:2454] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9336440ad04ede80187d1682fd6d1d in term 0.
I20260927 18:32:14.745573 1809 raft_consensus.cc:2454] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9336440ad04ede80187d1682fd6d1d in term 0.
I20260927 18:32:14.746148 1603 leader_election.cc:302] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b659659bb2c400288947436e49064fe, cc9336440ad04ede80187d1682fd6d1d; no voters:
I20260927 18:32:14.746948 2025 raft_consensus.cc:2781] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:14.747102 2025 raft_consensus.cc:482] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:14.747246 2025 raft_consensus.cc:3037] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.749327 2025 raft_consensus.cc:504] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.750039 2025 leader_election.cc:288] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [CANDIDATE]: Term 1 election: Requested vote from peers 487898b24c524b408fb7206fb52154fc (127.0.97.66:38881), 7b659659bb2c400288947436e49064fe (127.0.97.67:37957)
I20260927 18:32:14.750430 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e17d758d24e34bd2096d37a1fd36f" candidate_uuid: "cc9336440ad04ede80187d1682fd6d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487898b24c524b408fb7206fb52154fc"
I20260927 18:32:14.750548 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e17d758d24e34bd2096d37a1fd36f" candidate_uuid: "cc9336440ad04ede80187d1682fd6d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659659bb2c400288947436e49064fe"
I20260927 18:32:14.750633 1809 raft_consensus.cc:3037] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.750782 1952 raft_consensus.cc:3037] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:14.753738 1952 raft_consensus.cc:2454] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9336440ad04ede80187d1682fd6d1d in term 1.
I20260927 18:32:14.754143 1809 raft_consensus.cc:2454] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9336440ad04ede80187d1682fd6d1d in term 1.
I20260927 18:32:14.754259 1603 leader_election.cc:302] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b659659bb2c400288947436e49064fe, cc9336440ad04ede80187d1682fd6d1d; no voters:
I20260927 18:32:14.754523 2025 raft_consensus.cc:2781] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:14.755173 2025 raft_consensus.cc:686] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [term 1 LEADER]: Becoming Leader. State: Replica: cc9336440ad04ede80187d1682fd6d1d, State: Running, Role: LEADER
I20260927 18:32:14.755828 2025 consensus_queue.cc:237] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } }
I20260927 18:32:14.758069 2058 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: TxnStatusTablet state changed. Reason: New leader cc9336440ad04ede80187d1682fd6d1d. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.758534 2058 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:14.759274 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:32:14.759491 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:32:14.759483 1512 catalog_manager.cc:5495] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc9336440ad04ede80187d1682fd6d1d (127.0.97.65). New cstate: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:14.801754 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.793880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44037 (local address 127.0.0.1:51850)
0927 18:32:14.794098 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.794121 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.794274 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.794838 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.794846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.794861 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:14.795067 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.795080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.796548 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.796555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.798257 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.798272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.799678 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.799687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.799899 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.799972 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.801166 (+ 1194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.801188 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.801281 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.801438 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:32:14.801803 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.793727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.65:44037)
0927 18:32:14.793946 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.793953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.794207 (+ 254us) server_negotiation.cc:407] Connection header received
0927 18:32:14.794504 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.794509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.794639 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.794790 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:14.795224 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.796403 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.798435 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.799118 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.799261 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.800212 (+ 951us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.801020 (+ 808us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.801185 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.801323 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.801407 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:32:14.806842 1952 raft_consensus.cc:1261] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Refusing update from remote peer cc9336440ad04ede80187d1682fd6d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.806941 1809 raft_consensus.cc:1261] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Refusing update from remote peer cc9336440ad04ede80187d1682fd6d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.807760 2058 consensus_queue.cc:1035] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.808390 2025 consensus_queue.cc:1035] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.821099 2024 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: TxnStatusTablet state changed. Reason: New leader cc9336440ad04ede80187d1682fd6d1d. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.821460 2024 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.824787 2058 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.825052 2058 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:14.826804 2024 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.827064 2024 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 487898b24c524b408fb7206fb52154fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.829286 2023 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: TxnStatusTablet state changed. Reason: New leader cc9336440ad04ede80187d1682fd6d1d. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.829553 2023 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.830981 2058 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.831233 2058 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P cc9336440ad04ede80187d1682fd6d1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:14.836671 2023 tablet_replica.cc:439] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc9336440ad04ede80187d1682fd6d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 } } peers { permanent_uuid: "487898b24c524b408fb7206fb52154fc" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38881 } } peers { permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 } } }
I20260927 18:32:14.836925 2023 tablet_replica.cc:442] T 902e17d758d24e34bd2096d37a1fd36f P 7b659659bb2c400288947436e49064fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:14.856483 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.848458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43439 (local address 127.0.97.65:44037)
0927 18:32:14.848933 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.848940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.849860 (+ 920us) server_negotiation.cc:407] Connection header received
0927 18:32:14.850111 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.850118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.850209 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.850319 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:14.850871 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.852049 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.854210 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.855033 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.855164 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.855686 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.855827 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.855928 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.856084 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.856172 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260927 18:32:14.856581 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.848651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44037 (local address 127.0.97.66:43439)
0927 18:32:14.849730 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.849751 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.849936 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.850357 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.850362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.850375 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:14.850670 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.850681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.852188 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.852197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.854048 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.854066 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.855219 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.855226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.855415 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.855489 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.855941 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.855952 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.856037 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.856189 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":295,"threads_started":1}
I20260927 18:32:14.871343 1670 raft_consensus.cc:1261] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Refusing update from remote peer 487898b24c524b408fb7206fb52154fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.872414 1952 raft_consensus.cc:1261] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Refusing update from remote peer 487898b24c524b408fb7206fb52154fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:14.873139 2028 consensus_queue.cc:1035] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9336440ad04ede80187d1682fd6d1d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.873745 2024 consensus_queue.cc:1035] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b659659bb2c400288947436e49064fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:14.933174 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.921658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38881 (local address 127.0.97.65:56691)
0927 18:32:14.926381 (+ 4723us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:14.926402 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:14.926552 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:14.926990 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:14.926995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:14.927009 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:14.927369 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.927380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.928956 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.928961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.930814 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:14.930833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.931848 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.931855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:14.932024 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.932093 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.932518 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.932529 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:32:14.932620 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:32:14.932762 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4494,"thread_start_us":4323,"threads_started":1}
I20260927 18:32:14.934242 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.922029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56691 (local address 127.0.97.66:38881)
0927 18:32:14.922197 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.922204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:14.926509 (+ 4305us) server_negotiation.cc:407] Connection header received
0927 18:32:14.926715 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:14.926721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:14.926856 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:14.926967 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:14.927565 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.928785 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.930975 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:14.931659 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:14.931785 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:14.932288 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:14.932412 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:14.932501 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:14.933885 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0927 18:32:14.933971 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:32:14.974758 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:14.975018 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:14.979059 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb06577fd845429cba2f6492198e5291: Sending DeleteTablet for 3 replicas of tablet 170966a3e3394e9fbf38d396f7d33a3e
I20260927 18:32:14.979818 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34182
I20260927 18:32:14.980096 2042 tablet_replica.cc:330] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: stopping tablet replica
I20260927 18:32:14.980240 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38768
I20260927 18:32:14.980327 2042 raft_consensus.cc:2229] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:14.980509 2042 raft_consensus.cc:2258] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.980862 2041 tablet_replica.cc:330] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: stopping tablet replica
I20260927 18:32:14.981055 2041 raft_consensus.cc:2229] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:14.981393 2041 raft_consensus.cc:2258] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.981998 2042 ts_tablet_manager.cc:1905] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.982401 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 170966a3e3394e9fbf38d396f7d33a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:14 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49320
I20260927 18:32:14.983593 2041 ts_tablet_manager.cc:1905] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.985922 2040 tablet_replica.cc:330] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: stopping tablet replica
I20260927 18:32:14.986123 2040 raft_consensus.cc:2229] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:14.986334 2040 raft_consensus.cc:2258] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:14.989532 2042 ts_tablet_manager.cc:1918] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:14.989655 2042 log.cc:1196] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/170966a3e3394e9fbf38d396f7d33a3e
I20260927 18:32:14.990075 2042 ts_tablet_manager.cc:1939] T 170966a3e3394e9fbf38d396f7d33a3e P cc9336440ad04ede80187d1682fd6d1d: Deleting consensus metadata
I20260927 18:32:14.990731 2040 ts_tablet_manager.cc:1905] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:14.997745 2040 ts_tablet_manager.cc:1918] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:14.998153 2040 log.cc:1196] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/170966a3e3394e9fbf38d396f7d33a3e
I20260927 18:32:14.998623 2040 ts_tablet_manager.cc:1939] T 170966a3e3394e9fbf38d396f7d33a3e P 7b659659bb2c400288947436e49064fe: Deleting consensus metadata
I20260927 18:32:14.999434 2041 ts_tablet_manager.cc:1918] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:15.000624 1500 catalog_manager.cc:4841] TS 7b659659bb2c400288947436e49064fe (127.0.97.67:37957): tablet 170966a3e3394e9fbf38d396f7d33a3e (table test-table [id=c64bd0e946a2415bafb609be44ee0ffd]) successfully deleted
I20260927 18:32:15.000985 2041 log.cc:1196] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/170966a3e3394e9fbf38d396f7d33a3e
I20260927 18:32:15.001499 2041 ts_tablet_manager.cc:1939] T 170966a3e3394e9fbf38d396f7d33a3e P 487898b24c524b408fb7206fb52154fc: Deleting consensus metadata
I20260927 18:32:15.002164 1500 catalog_manager.cc:4841] TS cc9336440ad04ede80187d1682fd6d1d (127.0.97.65:44037): tablet 170966a3e3394e9fbf38d396f7d33a3e (table test-table [id=c64bd0e946a2415bafb609be44ee0ffd]) successfully deleted
I20260927 18:32:15.002676 1499 catalog_manager.cc:4841] TS 487898b24c524b408fb7206fb52154fc (127.0.97.66:38881): tablet 170966a3e3394e9fbf38d396f7d33a3e (table test-table [id=c64bd0e946a2415bafb609be44ee0ffd]) successfully deleted
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:15.019486 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:14.993247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.0.97.65:44037)
0927 18:32:14.993418 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:14.993425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:15.000275 (+ 6850us) server_negotiation.cc:407] Connection header received
0927 18:32:15.000673 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:15.000678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:15.000752 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:15.000817 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:15.002671 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.003853 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.005647 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.006823 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.006947 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:15.012017 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:15.012032 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:15.012051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:15.012094 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:15.016378 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.017256 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.017265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.017268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.017417 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.017841 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.017848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.017851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.018293 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:15.018630 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:15.018896 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:32:15.019125 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 18:32:15.020238 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51858
I20260927 18:32:15.036185 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:15.023320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.97.126:40537)
0927 18:32:15.023535 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:15.023541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:15.023709 (+ 168us) server_negotiation.cc:407] Connection header received
0927 18:32:15.023856 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:15.023860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:15.023919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:15.023994 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:15.025638 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.026513 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.027820 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.028217 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.028349 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:15.030672 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:15.030677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:15.030681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:15.030713 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:15.033563 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.034252 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.034259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.034263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.034389 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.034760 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.034767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.034770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.035032 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:15.035298 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:15.035604 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 18:32:15.035818 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 18:32:15.036787 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49132
I20260927 18:32:15.040131 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1584
I20260927 18:32:15.053459 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:15.041155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.97.65:44037)
0927 18:32:15.041314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:15.041323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:15.041370 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:32:15.041622 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:15.041627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:15.041704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:15.041836 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:15.042933 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.043669 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.044929 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.045448 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.045598 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:15.047811 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:15.047817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:15.047822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:15.047862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:15.050869 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.051511 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.051517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.051521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.051649 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.052017 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.052022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.052025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.052316 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:15.052590 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:15.052866 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:32:15.053082 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 18:32:15.054222 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51860
W20260927 18:32:15.054747 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:15.054953 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1584
I20260927 18:32:15.077888 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1724
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:15.094220 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:15.078700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.0.97.66:38881)
0927 18:32:15.078883 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:15.078888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:15.078905 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:32:15.079058 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:15.079062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:15.079141 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:15.079239 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:15.080642 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.081377 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.083157 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.083585 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.083682 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:15.088278 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:15.088290 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:15.088305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:15.088339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:15.091410 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.092131 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.092138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.092140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.092273 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.092680 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.092685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.092691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.093024 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:15.093378 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:15.093624 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:32:15.093871 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 18:32:15.094884 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39774
W20260927 18:32:15.095650 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:15.095803 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1724
I20260927 18:32:15.116946 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1863
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:15.131641 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:15.117745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.0.97.67:37957)
0927 18:32:15.117940 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:15.117948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:15.117969 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:15.118200 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:15.118205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:15.118267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:15.118381 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:15.119548 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.120288 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.121494 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:15.122041 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:15.122198 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:15.126416 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:15.126430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:15.126446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:15.126483 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:15.129114 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.129696 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.129703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.129705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.129997 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:15.130367 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:15.130371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:15.130373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:15.130713 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:15.130932 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:15.131208 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:32:15.131360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 18:32:15.132365 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60382
W20260927 18:32:15.132944 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:15.133091 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1863
I20260927 18:32:15.149828 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 1481
I20260927 18:32:15.150986 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:15.264380 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 1481
2026-09-27T18:32:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4566 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[2084](info): set up 2 sockets
Sep 27 18:32:15 dist-test-slave-n3p4 krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260927 18:32:16.361984 389 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:16 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:16.547664 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39667
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:16.864012 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:16.864310 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:16.864473 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:16.871891 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:16.871997 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:32:16.872092 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:16.872210 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:16.872290 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:16.872392 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:16.882198 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39667
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:16.883469 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:16.884799 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:16.896241 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:16.896207 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:16.896423 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:16.897107 2097 server_base.cc:1034] running on GCE node
I20260927 18:32:16.897913 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:16.899233 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:16.900439 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790533936900363 us; error 50 us; skew 500 ppm
I20260927 18:32:16.900722 2097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:16 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:16.903838 2097 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:16.905342 2097 webserver.cc:460] Webserver started at http://127.0.97.126:40809/ using document root <none> and password file <none>
I20260927 18:32:16.906107 2097 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:16.906212 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:16.906635 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:16.909675 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "254ca1cb80344f9dbad22c8d45f63972"
format_stamp: "Formatted at 2026-09-27 18:32:16 on dist-test-slave-n3p4"
server_key: "11e68008ec9df10d9f9cfe0ebd8e28f8"
server_key_iv: "542507ac3520e9401b73bc49e41511fd"
server_key_version: "encryptionkey@0"
I20260927 18:32:16.910503 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "254ca1cb80344f9dbad22c8d45f63972"
format_stamp: "Formatted at 2026-09-27 18:32:16 on dist-test-slave-n3p4"
server_key: "11e68008ec9df10d9f9cfe0ebd8e28f8"
server_key_iv: "542507ac3520e9401b73bc49e41511fd"
server_key_version: "encryptionkey@0"
I20260927 18:32:16.915705 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:16.919137 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:16.920399 2097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:16.920559 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "254ca1cb80344f9dbad22c8d45f63972"
format_stamp: "Formatted at 2026-09-27 18:32:16 on dist-test-slave-n3p4"
server_key: "11e68008ec9df10d9f9cfe0ebd8e28f8"
server_key_iv: "542507ac3520e9401b73bc49e41511fd"
server_key_version: "encryptionkey@0"
I20260927 18:32:16.920773 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:16.934476 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:16.938454 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:16.938748 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:16.960080 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:39667
I20260927 18:32:16.960098 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:39667 every 8 connection(s)
I20260927 18:32:16.961431 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:16.964795 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2097
I20260927 18:32:16.965023 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:16.965672 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:16.965864 389 external_mini_cluster.cc:1428] Setting key 3bccaa22c6b7db27b5b6d42497a402d2
I20260927 18:32:16.980780 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Bootstrap starting.
Sep 27 18:32:16 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:16.984836 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:16.986194 2174 log.cc:824] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:16.989607 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: No bootstrap required, opened a new log
I20260927 18:32:16.990262 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:16.968063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.0.97.126:39667)
0927 18:32:16.968932 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:16.968943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:16.968988 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:32:16.970764 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:16.970788 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:16.971192 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:16.971526 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:16.973353 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:16.974689 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:16.976453 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:16.977134 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:16.977264 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:16.981925 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:16.981936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:16.981956 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:16.982000 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:16.985861 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:16.986527 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:16.986536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:16.986544 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:16.986774 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:16.987208 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:16.987211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:16.987214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:16.987891 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:16.988146 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:16.988509 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 18:32:16.988934 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":386,"threads_started":1}
I20260927 18:32:16.996953 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } }
I20260927 18:32:16.997359 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:16.997468 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254ca1cb80344f9dbad22c8d45f63972, State: Initialized, Role: FOLLOWER
I20260927 18:32:16.998208 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } }
I20260927 18:32:16.998441 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:16.998564 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:16.998734 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:17.000810 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } }
I20260927 18:32:17.001308 2174 leader_election.cc:302] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254ca1cb80344f9dbad22c8d45f63972; no voters:
I20260927 18:32:17.001933 2174 leader_election.cc:288] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:17.002041 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:17.002912 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [term 1 LEADER]: Becoming Leader. State: Replica: 254ca1cb80344f9dbad22c8d45f63972, State: Running, Role: LEADER
I20260927 18:32:17.003497 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } }
I20260927 18:32:17.004062 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:17.006246 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254ca1cb80344f9dbad22c8d45f63972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } } }
I20260927 18:32:17.006631 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:17.007028 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254ca1cb80344f9dbad22c8d45f63972. Latest consensus state: current_term: 1 leader_uuid: "254ca1cb80344f9dbad22c8d45f63972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ca1cb80344f9dbad22c8d45f63972" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39667 } } }
I20260927 18:32:17.007300 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:17.008841 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:17.014025 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:17.022775 2187 catalog_manager.cc:1347] Generated new cluster ID: 6626131cc90945e88999cf637f649c75
I20260927 18:32:17.022895 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:17.042116 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:17.043994 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:17.056047 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Generated new TSK 0
I20260927 18:32:17.057093 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:17.101158 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39667
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:17.435850 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:17.436167 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:17.436271 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:17.443982 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:17.444113 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:17.444199 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:32:17.444253 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:17.444321 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:17.454356 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:39667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:17.455634 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:17.456943 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:17.470280 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:17.470254 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:17.470615 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:17.471294 2201 server_base.cc:1034] running on GCE node
I20260927 18:32:17.471892 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:17.472767 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:17.473994 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790533937473950 us; error 57 us; skew 500 ppm
I20260927 18:32:17.474319 2201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:17 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:17.477403 2201 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:17.478997 2201 webserver.cc:460] Webserver started at http://127.0.97.65:33125/ using document root <none> and password file <none>
I20260927 18:32:17.479671 2201 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:17.479823 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:17.480126 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:17.483099 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "ec40c6729e344df49eec9e089ba5c79b"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "133a5ee8be5e35438ee61f2ace1e2076"
server_key_iv: "edca316608ae600aeb203ef3bcdc76c4"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.483871 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "ec40c6729e344df49eec9e089ba5c79b"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "133a5ee8be5e35438ee61f2ace1e2076"
server_key_iv: "edca316608ae600aeb203ef3bcdc76c4"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.489025 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:17.492597 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:17.494027 2201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:17.494156 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "ec40c6729e344df49eec9e089ba5c79b"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "133a5ee8be5e35438ee61f2ace1e2076"
server_key_iv: "edca316608ae600aeb203ef3bcdc76c4"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.494320 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:17.508704 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:17.513340 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:17.513693 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:17.516551 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:17.516700 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:17.516812 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:17.516913 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:32:17 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:17.542958 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:17.518102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.65:45831)
0927 18:32:17.518981 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:17.519032 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:17.521322 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:17.522332 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:17.522341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:17.523182 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:17.524131 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:17.524155 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.526017 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.526024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:17.528145 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:17.528186 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.529297 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.529304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:17.529579 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:17.530473 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:17.530496 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:17.533293 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:17.537328 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:17.537341 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:17.537358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:17.538106 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:17.538595 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:17.538601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:17.538606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:17.538969 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:17.539737 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:17.539747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:17.540269 (+ 522us) client_negotiation.cc:769] Sending connection context
0927 18:32:17.540682 (+ 413us) client_negotiation.cc:240] Negotiation successful
0927 18:32:17.541206 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260927 18:32:17.543720 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:17.517415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45831 (local address 127.0.97.126:39667)
0927 18:32:17.517868 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:17.517874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:17.519164 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 18:32:17.521675 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:17.521681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:17.521772 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:17.521907 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:17.524419 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.525813 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.528341 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.529013 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.529158 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:17.533572 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:17.533581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:17.533588 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:17.533636 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:17.537169 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:17.538258 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:17.538266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:17.538270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:17.538426 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:17.539132 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:17.539137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:17.539141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:17.539446 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:17.539637 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:17.543046 (+ 3409us) server_negotiation.cc:299] Negotiation successful
0927 18:32:17.543313 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
I20260927 18:32:17.560725 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42837
I20260927 18:32:17.560793 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42837 every 8 connection(s)
I20260927 18:32:17.562570 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:17.565142 2337 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39667
I20260927 18:32:17.565384 2337 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:17.566152 2337 heartbeater.cc:511] Master 127.0.97.126:39667 requested a full tablet report, sending...
I20260927 18:32:17.566761 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:17.551036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41473 (local address 127.0.97.126:39667)
0927 18:32:17.551244 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:17.551253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:17.551864 (+ 611us) server_negotiation.cc:407] Connection header received
0927 18:32:17.552136 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:17.552141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:17.552214 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:17.552315 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:17.553189 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.554250 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.556789 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.557330 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.557477 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:17.559709 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:17.559716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:17.559721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:17.559764 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:17.563552 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:17.564595 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:17.564600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:17.564604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:17.564769 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:17.565174 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:17.565182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:17.565186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:17.565563 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:17.565717 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:17.566191 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 18:32:17.566420 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:32:17.566912 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:17.551068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.65:41473)
0927 18:32:17.551750 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:17.551775 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:17.551984 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:17.552339 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:17.552344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:17.552793 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:17.553053 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:17.553062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.554388 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.554394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:17.556347 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:17.556383 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:17.557545 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:17.557554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:17.557739 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:17.558608 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:17.558636 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:17.559503 (+ 867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:17.563767 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:17.563777 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:17.563782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:17.564388 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:17.564879 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:17.564883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:17.564886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:17.565067 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:17.565684 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:17.565689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:17.565921 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 18:32:17.566081 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 18:32:17.566440 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":358,"threads_started":1}
I20260927 18:32:17.568562 2129 ts_manager.cc:194] Registered new tserver with Master: ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837)
I20260927 18:32:17.569581 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2201
I20260927 18:32:17.569814 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:17.570533 389 external_mini_cluster.cc:1428] Setting key 391074c294741f69a4cc3500e4340a5c
I20260927 18:32:17.570591 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:45831
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:17.611105 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39667
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:17.930469 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:17.930728 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:17.930827 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:17.938364 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:17.938469 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:17.938553 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:32:17.938611 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:17.938675 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:17.948489 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:39667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:17.949702 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:17.950932 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:17.964543 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:17.964576 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:17.964968 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:17.965852 2341 server_base.cc:1034] running on GCE node
I20260927 18:32:17.966523 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:17.967458 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:17.968711 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790533937968663 us; error 43 us; skew 500 ppm
I20260927 18:32:17.969019 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:17 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:17.972126 2341 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:17.973526 2341 webserver.cc:460] Webserver started at http://127.0.97.66:37017/ using document root <none> and password file <none>
I20260927 18:32:17.974262 2341 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:17.974399 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:17.974699 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:17.977576 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "e204f78cb000478dbfd18f7f65ff71f2"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "6faabdafa4edeb1e4ce507943f1d0301"
server_key_iv: "914aa7a73df782bbadb29de83f13b232"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.978448 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "e204f78cb000478dbfd18f7f65ff71f2"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "6faabdafa4edeb1e4ce507943f1d0301"
server_key_iv: "914aa7a73df782bbadb29de83f13b232"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.983422 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:17.986857 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:17.988221 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:17.988405 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "e204f78cb000478dbfd18f7f65ff71f2"
format_stamp: "Formatted at 2026-09-27 18:32:17 on dist-test-slave-n3p4"
server_key: "6faabdafa4edeb1e4ce507943f1d0301"
server_key_iv: "914aa7a73df782bbadb29de83f13b232"
server_key_version: "encryptionkey@0"
I20260927 18:32:17.988641 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:18.012732 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:18.020470 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:18.020798 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:18.023682 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:18.023849 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:32:18.023983 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:18.024111 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:18.053292 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.024240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51001 (local address 127.0.97.126:39667)
0927 18:32:18.024433 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.024441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.026256 (+ 1815us) server_negotiation.cc:407] Connection header received
0927 18:32:18.029452 (+ 3196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.029459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.029589 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.029735 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.032537 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.033393 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.036960 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.037749 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.037932 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.044230 (+ 6298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.044237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.044244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.044290 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.048220 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.049533 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.049541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.049546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.049746 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.050433 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.050438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.050441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.050855 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.050993 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.052554 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.052811 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:32:18.054687 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.024901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.66:51001)
0927 18:32:18.026024 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.026097 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.028992 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.030090 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.030109 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.031134 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:18.032301 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.032347 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.033575 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.033584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.036773 (+ 3189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.036794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.038083 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.038093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.038539 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.039489 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:18.039536 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:18.043820 (+ 4284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:18.048425 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.048438 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.048470 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.049356 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.049896 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.049901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.049905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.050317 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.050982 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:18.050991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:18.051587 (+ 596us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.051949 (+ 362us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.052582 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":410,"threads_started":1}
I20260927 18:32:18.081189 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44597
I20260927 18:32:18.081235 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44597 every 8 connection(s)
I20260927 18:32:18.082182 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.063233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.66:55341)
0927 18:32:18.064497 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.064518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.064715 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.065288 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.065293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.066172 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:18.066557 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.066572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.068754 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.068758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.070614 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.070627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.071874 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.071884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.072165 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.073174 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:18.073208 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:18.074649 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:18.078838 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.078844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.078851 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.079447 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.079973 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.079978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.079981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.080161 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.080873 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:18.080878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:18.081116 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.081287 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.081683 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":552,"threads_started":1}
I20260927 18:32:18.082561 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.063239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55341 (local address 127.0.97.126:39667)
0927 18:32:18.063422 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.063430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.064601 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 18:32:18.064873 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.064880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.064985 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.065061 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.066735 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.067843 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.070810 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.071653 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.071802 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.074858 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.074866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.074872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.074916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.078679 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.079679 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.079684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.079688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.079835 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.080332 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.080338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.080343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.080737 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.081889 (+ 1152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.082028 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.082255 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:32:18.083078 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:18.085278 2476 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39667
I20260927 18:32:18.086148 2476 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:18.087070 2476 heartbeater.cc:511] Master 127.0.97.126:39667 requested a full tablet report, sending...
I20260927 18:32:18.088713 2129 ts_manager.cc:194] Registered new tserver with Master: e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:18.089273 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2341
I20260927 18:32:18.089457 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:18.089887 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51001
I20260927 18:32:18.089986 389 external_mini_cluster.cc:1428] Setting key 458097858ec7c13466cf2dbe1537292b
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:18.136826 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39667
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:18.455582 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:18.455899 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:18.456080 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:18.463512 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:18.463662 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:18.463824 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:32:18.463910 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:32:18.464025 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:18.473743 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:39667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:18.475135 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:18.476509 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:18.489395 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:18.489405 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:18.489482 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:18.490549 2480 server_base.cc:1034] running on GCE node
I20260927 18:32:18.491182 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:18.492225 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:18.493422 2480 hybrid_clock.cc:648] HybridClock initialized: now 1790533938493388 us; error 35 us; skew 500 ppm
I20260927 18:32:18.493826 2480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:18.496877 2480 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:18.498523 2480 webserver.cc:460] Webserver started at http://127.0.97.67:45343/ using document root <none> and password file <none>
I20260927 18:32:18.499219 2480 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:18.499364 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:18.499683 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:18.502689 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "420f1328e7584dde8b57bbb7a815c005"
format_stamp: "Formatted at 2026-09-27 18:32:18 on dist-test-slave-n3p4"
server_key: "b4957f46cea2356a3113b5eef43fd4cf"
server_key_iv: "a3d112d3ed79562465a2a0fe1f086411"
server_key_version: "encryptionkey@0"
I20260927 18:32:18.503453 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "420f1328e7584dde8b57bbb7a815c005"
format_stamp: "Formatted at 2026-09-27 18:32:18 on dist-test-slave-n3p4"
server_key: "b4957f46cea2356a3113b5eef43fd4cf"
server_key_iv: "a3d112d3ed79562465a2a0fe1f086411"
server_key_version: "encryptionkey@0"
I20260927 18:32:18.508868 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:18.512765 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:18.514283 2480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:32:18.514456 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "420f1328e7584dde8b57bbb7a815c005"
format_stamp: "Formatted at 2026-09-27 18:32:18 on dist-test-slave-n3p4"
server_key: "b4957f46cea2356a3113b5eef43fd4cf"
server_key_iv: "a3d112d3ed79562465a2a0fe1f086411"
server_key_version: "encryptionkey@0"
I20260927 18:32:18.514665 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:18.528851 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:18.532965 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:18.533249 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:18.535779 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:18.535909 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:18.536051 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:18.536161 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:18.557056 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.536535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:48385 (local address 127.0.97.126:39667)
0927 18:32:18.536773 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.536783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.538082 (+ 1299us) server_negotiation.cc:407] Connection header received
0927 18:32:18.540088 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.540097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.540196 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.540344 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.542560 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.543452 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.545048 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.545724 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.545893 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.550195 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.550204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.550212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.550258 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.553193 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.554233 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.554241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.554248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.554443 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.555003 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.555009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.555012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.555437 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.555594 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.556405 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.556669 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:18.558408 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.537030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.67:48385)
0927 18:32:18.537906 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.537956 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.539806 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.540643 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.540655 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.541473 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:18.542386 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.542406 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.543614 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.543621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.544908 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.544920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.546026 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.546036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.546381 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.547068 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:18.547095 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:18.549964 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:18.553330 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.553340 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.553364 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.554077 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.554576 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.554582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.554586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.554880 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.555561 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:18.555574 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:18.556006 (+ 432us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.556273 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.556773 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":366,"threads_started":1}
I20260927 18:32:18.573612 2337 heartbeater.cc:503] Master 127.0.97.126:39667 was elected leader, sending a full tablet report...
I20260927 18:32:18.592772 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.572657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.97.67:35951)
0927 18:32:18.574131 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.574151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.574590 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.575360 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.575368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.576185 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:18.576894 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.576918 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.578428 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.578434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.580637 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.580654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.581991 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.582005 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.582305 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.583166 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:18.583196 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:18.584839 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:18.589226 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.589236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.589244 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.590028 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.590553 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.590561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.590568 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.590751 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.591492 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:18.591500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:18.591766 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.591960 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.592275 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":486,"threads_started":1}
I20260927 18:32:18.592799 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.573164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35951 (local address 127.0.97.126:39667)
0927 18:32:18.573351 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.573356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.574248 (+ 892us) server_negotiation.cc:407] Connection header received
0927 18:32:18.574789 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.574797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.574891 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.574965 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.577105 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.578221 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.580843 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.581612 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.581749 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.585091 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.585097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.585101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.585137 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.589048 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.590201 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.590209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.590216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.590400 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.590909 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.590914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.590919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.591341 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.591526 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.592087 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.592328 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:32:18.594645 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37831
I20260927 18:32:18.594738 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37831 every 8 connection(s)
I20260927 18:32:18.595979 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:18.598119 2615 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39667
I20260927 18:32:18.598321 2615 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:18.598932 2615 heartbeater.cc:511] Master 127.0.97.126:39667 requested a full tablet report, sending...
I20260927 18:32:18.600191 2129 ts_manager.cc:194] Registered new tserver with Master: 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831)
I20260927 18:32:18.601023 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:48385
I20260927 18:32:18.604348 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2480
I20260927 18:32:18.604537 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:18.604933 389 external_mini_cluster.cc:1428] Setting key 9ebf556ce4881f401b399fc4de15fee5
I20260927 18:32:18.608085 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:18.654887 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.640951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.97.126:39667)
0927 18:32:18.641140 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.641145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.641475 (+ 330us) server_negotiation.cc:407] Connection header received
0927 18:32:18.641687 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.641691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.641838 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.641945 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.643289 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.644000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.645219 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.645715 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.645860 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.649836 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.649846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.649855 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.649907 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.652393 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.653055 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.653058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.653061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.653175 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.653504 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.653507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.653508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.653886 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.654063 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.654352 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.654513 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:32:18.659245 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:18.667309 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:18.698168 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.679850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.0.1:36794)
0927 18:32:18.687618 (+ 7768us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.687658 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.687842 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.689517 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.689545 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.689601 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:18.690137 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.690155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.692842 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.692853 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.695614 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.695627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.697058 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.697065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.697286 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.697356 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.697484 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.697684 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7486,"thread_start_us":8523,"threads_started":1}
I20260927 18:32:18.699036 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.679967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.0.97.65:42837)
0927 18:32:18.688364 (+ 8397us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.688378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.688415 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:32:18.688550 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.688562 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.688916 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.689183 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:18.690434 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.692612 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.695796 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.696858 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.697039 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.697279 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.698460 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.698718 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8176,"thread_start_us":8005,"threads_started":1}
I20260927 18:32:18.703742 2267 tablet_service.cc:1467] Processing CreateTablet for tablet e970eb340c064916bcf5c22ff4a77278 (DEFAULT_TABLE table=test-table [id=2f631d91d0ef46de958677672c11c006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:18.705134 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e970eb340c064916bcf5c22ff4a77278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.706426 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.679916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44597 (local address 127.0.0.1:57470)
0927 18:32:18.688690 (+ 8774us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.688709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.688827 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.698205 (+ 9378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.698212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.698242 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:18.698642 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.698658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.701922 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.701926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.703687 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.703701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.705589 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.705595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.705767 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.705827 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.705916 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.706041 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8641,"thread_start_us":7273,"threads_started":1}
I20260927 18:32:18.707044 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.680468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.0.97.66:44597)
0927 18:32:18.695110 (+ 14642us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.695123 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.695161 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:32:18.697416 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.697426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.697731 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.697974 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:18.698832 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.700249 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.703895 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.705379 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.705516 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.705745 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.706630 (+ 885us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.706798 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14432,"thread_start_us":14295,"threads_started":1}
I20260927 18:32:18.710362 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.689034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37831 (local address 127.0.0.1:51442)
0927 18:32:18.695745 (+ 6711us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.695767 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.695891 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.703459 (+ 7568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.703464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.703488 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:18.703923 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.703934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.705921 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.705930 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.707540 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.707556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.709583 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.709589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.709762 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.709823 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.709908 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.710029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6575,"thread_start_us":6938,"threads_started":1}
I20260927 18:32:18.711025 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.689040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.0.97.67:37831)
0927 18:32:18.702444 (+ 13404us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.702453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.702486 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:32:18.702572 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.702580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.702811 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.702989 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:18.704152 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.705734 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.707691 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.709110 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.709233 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.709444 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.710559 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.710720 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13229,"thread_start_us":14177,"threads_started":1}
I20260927 18:32:18.710821 2406 tablet_service.cc:1467] Processing CreateTablet for tablet e970eb340c064916bcf5c22ff4a77278 (DEFAULT_TABLE table=test-table [id=2f631d91d0ef46de958677672c11c006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:18.712049 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e970eb340c064916bcf5c22ff4a77278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.718457 2634 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Bootstrap starting.
I20260927 18:32:18.717825 2549 tablet_service.cc:1467] Processing CreateTablet for tablet e970eb340c064916bcf5c22ff4a77278 (DEFAULT_TABLE table=test-table [id=2f631d91d0ef46de958677672c11c006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:18.719599 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e970eb340c064916bcf5c22ff4a77278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.721490 2634 tablet_bootstrap.cc:654] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.722681 2634 log.cc:824] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:18.725536 2634 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: No bootstrap required, opened a new log
I20260927 18:32:18.725871 2634 ts_tablet_manager.cc:1397] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 18:32:18.727497 2636 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Bootstrap starting.
I20260927 18:32:18.730924 2636 tablet_bootstrap.cc:654] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.732090 2636 log.cc:824] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:18.733086 2637 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Bootstrap starting.
I20260927 18:32:18.736975 2637 tablet_bootstrap.cc:654] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.736701 2634 raft_consensus.cc:348] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.737176 2634 raft_consensus.cc:374] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.737282 2634 raft_consensus.cc:729] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.737918 2634 consensus_queue.cc:260] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.738376 2637 log.cc:824] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:18.738910 2636 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: No bootstrap required, opened a new log
I20260927 18:32:18.739164 2636 ts_tablet_manager.cc:1397] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260927 18:32:18.739727 2634 ts_tablet_manager.cc:1428] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 18:32:18.744262 2637 tablet_bootstrap.cc:492] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: No bootstrap required, opened a new log
I20260927 18:32:18.744565 2637 ts_tablet_manager.cc:1397] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 18:32:18.747857 2636 raft_consensus.cc:348] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.748204 2636 raft_consensus.cc:374] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.748299 2636 raft_consensus.cc:729] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.748970 2636 consensus_queue.cc:260] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.750075 2476 heartbeater.cc:503] Master 127.0.97.126:39667 was elected leader, sending a full tablet report...
I20260927 18:32:18.750705 2636 ts_tablet_manager.cc:1428] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 18:32:18.753224 2637 raft_consensus.cc:348] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.753618 2637 raft_consensus.cc:374] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.753743 2637 raft_consensus.cc:729] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.754478 2637 consensus_queue.cc:260] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.755566 2615 heartbeater.cc:503] Master 127.0.97.126:39667 was elected leader, sending a full tablet report...
I20260927 18:32:18.756276 2637 ts_tablet_manager.cc:1428] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Time spent starting tablet: real 0.012s user 0.002s sys 0.011s
W20260927 18:32:18.818130 2338 tablet.cc:2369] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:18.825127 2642 raft_consensus.cc:482] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:18.825475 2642 raft_consensus.cc:504] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.827272 2642 leader_election.cc:288] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:18.832902 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.827731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.97.67:41329)
0927 18:32:18.827954 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.827980 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.828152 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.828488 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.828493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.828519 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:18.828765 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.828773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.829768 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.829822 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.831105 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.831119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.832196 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.832202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.832387 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.832410 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.832515 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.832641 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:32:18.832902 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.827849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41329 (local address 127.0.97.65:42837)
0927 18:32:18.828020 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.828028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.828049 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:18.828302 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.828307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.828380 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.828474 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:18.828905 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.829620 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.831268 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.832038 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.832184 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.832291 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.832540 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.832671 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:32:18.834254 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.827808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:39111 (local address 127.0.97.66:44597)
0927 18:32:18.828161 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.828167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.829059 (+ 892us) server_negotiation.cc:407] Connection header received
0927 18:32:18.829297 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.829301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.829369 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.829471 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:18.830100 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.830818 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.832385 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.833329 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.833460 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.833573 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.833961 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.834066 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260927 18:32:18.834106 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e970eb340c064916bcf5c22ff4a77278" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b" is_pre_election: true
I20260927 18:32:18.834362 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.828146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44597 (local address 127.0.97.67:39111)
0927 18:32:18.828940 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.828954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.829135 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.829482 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.829486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.829507 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:18.829932 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.829941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.830951 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.830957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.832232 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.832243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.833562 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.833570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.833751 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.833783 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.833907 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.834031 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":489,"threads_started":1}
I20260927 18:32:18.834687 2287 raft_consensus.cc:2454] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
W20260927 18:32:18.835268 2477 tablet.cc:2369] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:18.835492 2498 leader_election.cc:302] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:18.835462 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e970eb340c064916bcf5c22ff4a77278" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2" is_pre_election: true
I20260927 18:32:18.835897 2642 raft_consensus.cc:2781] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:18.835994 2426 raft_consensus.cc:2454] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
I20260927 18:32:18.836186 2642 raft_consensus.cc:482] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:18.836301 2642 raft_consensus.cc:3037] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:18.838965 2642 raft_consensus.cc:504] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.839702 2642 leader_election.cc:288] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Requested vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:18.840174 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e970eb340c064916bcf5c22ff4a77278" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b"
I20260927 18:32:18.840174 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e970eb340c064916bcf5c22ff4a77278" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2"
I20260927 18:32:18.840385 2426 raft_consensus.cc:3037] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:18.840386 2287 raft_consensus.cc:3037] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:18.843591 2287 raft_consensus.cc:2454] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:18.843587 2426 raft_consensus.cc:2454] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:18.844036 2499 leader_election.cc:302] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, e204f78cb000478dbfd18f7f65ff71f2; no voters:
I20260927 18:32:18.844395 2642 raft_consensus.cc:2781] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:18.844854 2642 raft_consensus.cc:686] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Becoming Leader. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Running, Role: LEADER
I20260927 18:32:18.845332 2642 consensus_queue.cc:237] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
W20260927 18:32:18.847590 2616 tablet.cc:2369] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:18.850423 2129 catalog_manager.cc:5495] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "420f1328e7584dde8b57bbb7a815c005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } health_report { overall_health: HEALTHY } } }
Sep 27 18:32:18 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:18.926903 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.912275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.0.97.126:39667)
0927 18:32:18.912479 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:18.912491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:18.913123 (+ 632us) server_negotiation.cc:407] Connection header received
0927 18:32:18.913253 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:18.913257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:18.913320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:18.913410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:18.914721 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.915502 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.916789 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.917310 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.917414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.921042 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:18.921047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:18.921051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:18.921083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:18.923994 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.924689 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.924693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.924695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.924819 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:18.925366 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:18.925372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:18.925378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:18.925726 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:18.925922 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:18.926331 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 18:32:18.926606 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":59}
I20260927 18:32:18.926908 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:18.912390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39667 (local address 127.0.0.1:40560)
0927 18:32:18.913003 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:18.913020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:18.913152 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:18.913486 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:18.913490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:18.914262 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:18.914568 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.914578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.915642 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.915647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.916638 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:18.916649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:18.917491 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:18.917496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:18.917664 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:18.918427 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:18.918489 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:18.920860 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:18.924112 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.924122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.924127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.924565 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.924954 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:18.924957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:18.924960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:18.925174 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:18.925901 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:18.925908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:18.926076 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 18:32:18.926224 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:32:18.926448 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":352,"threads_started":1}
I20260927 18:32:18.931985 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:32:18.932983 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:18.939682 2267 tablet_service.cc:1467] Processing CreateTablet for tablet f26ea12b641b4f348ced0f9bb00dca32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d74e2d94ef2d481d9134cdd32c0703cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:18.939874 2406 tablet_service.cc:1467] Processing CreateTablet for tablet f26ea12b641b4f348ced0f9bb00dca32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d74e2d94ef2d481d9134cdd32c0703cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:18.940267 2549 tablet_service.cc:1467] Processing CreateTablet for tablet f26ea12b641b4f348ced0f9bb00dca32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d74e2d94ef2d481d9134cdd32c0703cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:32:18.940775 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26ea12b641b4f348ced0f9bb00dca32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.940809 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26ea12b641b4f348ced0f9bb00dca32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.941112 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26ea12b641b4f348ced0f9bb00dca32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:18.947216 2636 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: Bootstrap starting.
I20260927 18:32:18.948213 2637 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: Bootstrap starting.
I20260927 18:32:18.948256 2634 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: Bootstrap starting.
I20260927 18:32:18.948902 2636 tablet_bootstrap.cc:654] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.949587 2637 tablet_bootstrap.cc:654] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.950112 2634 tablet_bootstrap.cc:654] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:18.952243 2637 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: No bootstrap required, opened a new log
I20260927 18:32:18.952399 2637 ts_tablet_manager.cc:1397] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 18:32:18.952600 2636 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: No bootstrap required, opened a new log
I20260927 18:32:18.952769 2636 ts_tablet_manager.cc:1397] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260927 18:32:18.953017 2634 tablet_bootstrap.cc:492] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: No bootstrap required, opened a new log
I20260927 18:32:18.953181 2634 ts_tablet_manager.cc:1397] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:18.953301 2637 raft_consensus.cc:348] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.953585 2637 raft_consensus.cc:374] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.953487 2636 raft_consensus.cc:348] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.953689 2637 raft_consensus.cc:729] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.953773 2636 raft_consensus.cc:374] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.953907 2636 raft_consensus.cc:729] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.953967 2637 consensus_queue.cc:260] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.954046 2634 raft_consensus.cc:348] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.954131 2636 consensus_queue.cc:260] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.954278 2634 raft_consensus.cc:374] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:18.954370 2634 raft_consensus.cc:729] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Initialized, Role: FOLLOWER
I20260927 18:32:18.954455 2642 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:18.954691 2637 ts_tablet_manager.cc:1428] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:32:18.954753 2642 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:18.954555 2641 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:18.954675 2634 consensus_queue.cc:260] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:18.954882 2641 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:18.955319 2634 ts_tablet_manager.cc:1428] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:18.955612 2639 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:18.955967 2639 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:18.956902 2636 ts_tablet_manager.cc:1428] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 18:32:19.209551 2639 raft_consensus.cc:482] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:19.209904 2639 raft_consensus.cc:504] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.211594 2639 leader_election.cc:288] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597), 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831)
I20260927 18:32:19.218590 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.212161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45085 (local address 127.0.97.66:44597)
0927 18:32:19.212419 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.212430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.213013 (+ 583us) server_negotiation.cc:407] Connection header received
0927 18:32:19.213418 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.213427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.213538 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.213610 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:19.214302 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.215049 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.216825 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.217592 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.217738 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.217864 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.218185 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.218306 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:19.218675 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.211960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44597 (local address 127.0.97.65:45085)
0927 18:32:19.212906 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.212927 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.213224 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.213864 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.213867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.213887 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:19.214150 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.214157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.215172 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.215180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.216681 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.216692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.217812 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.217819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.218000 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.218062 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.218162 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.218288 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":24,"spinlock_wait_cycles":67072,"thread_start_us":794,"threads_started":1}
I20260927 18:32:19.219309 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26ea12b641b4f348ced0f9bb00dca32" candidate_uuid: "ec40c6729e344df49eec9e089ba5c79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2" is_pre_election: true
I20260927 18:32:19.219547 2426 raft_consensus.cc:2454] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec40c6729e344df49eec9e089ba5c79b in term 0.
I20260927 18:32:19.219808 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.212205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37831 (local address 127.0.97.65:60755)
0927 18:32:19.213022 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.213050 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.213218 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.214258 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.214263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.214287 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:19.214679 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.214687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.215755 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.215762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.217343 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.217356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.218945 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.218952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.219141 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.219182 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.219260 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.219350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":43,"spinlock_wait_cycles":1408,"thread_start_us":754,"threads_started":1}
I20260927 18:32:19.220027 2220 leader_election.cc:302] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e204f78cb000478dbfd18f7f65ff71f2, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:19.220405 2639 raft_consensus.cc:2781] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:19.220546 2639 raft_consensus.cc:482] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:19.220645 2639 raft_consensus.cc:3037] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.220736 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.213317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60755 (local address 127.0.97.67:37831)
0927 18:32:19.213982 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.213987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.214003 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:32:19.214058 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.214061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.214127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.214247 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:19.214838 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.215605 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.217504 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.218672 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.220081 (+ 1409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.220199 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.220253 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.220392 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":397,"threads_started":1}
I20260927 18:32:19.221546 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26ea12b641b4f348ced0f9bb00dca32" candidate_uuid: "ec40c6729e344df49eec9e089ba5c79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420f1328e7584dde8b57bbb7a815c005" is_pre_election: true
I20260927 18:32:19.222005 2570 raft_consensus.cc:2454] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec40c6729e344df49eec9e089ba5c79b in term 0.
I20260927 18:32:19.223609 2639 raft_consensus.cc:504] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.224352 2639 leader_election.cc:288] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [CANDIDATE]: Term 1 election: Requested vote from peers e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597), 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831)
I20260927 18:32:19.224958 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26ea12b641b4f348ced0f9bb00dca32" candidate_uuid: "ec40c6729e344df49eec9e089ba5c79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420f1328e7584dde8b57bbb7a815c005"
I20260927 18:32:19.224941 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26ea12b641b4f348ced0f9bb00dca32" candidate_uuid: "ec40c6729e344df49eec9e089ba5c79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2"
I20260927 18:32:19.225171 2570 raft_consensus.cc:3037] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.225224 2426 raft_consensus.cc:3037] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.228552 2426 raft_consensus.cc:2454] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec40c6729e344df49eec9e089ba5c79b in term 1.
I20260927 18:32:19.228567 2570 raft_consensus.cc:2454] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec40c6729e344df49eec9e089ba5c79b in term 1.
I20260927 18:32:19.228984 2220 leader_election.cc:302] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e204f78cb000478dbfd18f7f65ff71f2, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:19.229303 2639 raft_consensus.cc:2781] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:19.229708 2639 raft_consensus.cc:686] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [term 1 LEADER]: Becoming Leader. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Running, Role: LEADER
I20260927 18:32:19.230237 2639 consensus_queue.cc:237] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.233547 2658 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: TxnStatusTablet state changed. Reason: New leader ec40c6729e344df49eec9e089ba5c79b. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.234041 2658 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:19.234829 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:32:19.235131 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:32:19.235785 2127 catalog_manager.cc:5495] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec40c6729e344df49eec9e089ba5c79b (127.0.97.65). New cstate: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:19.321836 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.315678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.0.97.65:42837)
0927 18:32:19.315860 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.315864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.315922 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:32:19.316226 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.316229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.316308 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.316416 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.316803 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.317571 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.319546 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.320084 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.320184 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.320691 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.321225 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.321338 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.321472 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.321537 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:32:19.321928 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.315571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.0.1:36796)
0927 18:32:19.315832 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.315853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.315998 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.316463 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.316468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.316481 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:19.316679 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.316689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.317688 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.317693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.319388 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.319402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.320231 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.320238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.320429 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.320498 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.321330 (+ 832us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.321349 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.321444 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.321594 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:32:19.329452 2570 raft_consensus.cc:1261] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Refusing update from remote peer ec40c6729e344df49eec9e089ba5c79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:19.329627 2426 raft_consensus.cc:1261] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Refusing update from remote peer ec40c6729e344df49eec9e089ba5c79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:19.330495 2658 consensus_queue.cc:1035] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.330940 2639 consensus_queue.cc:1035] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.342846 2642 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: TxnStatusTablet state changed. Reason: New leader ec40c6729e344df49eec9e089ba5c79b. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.343146 2642 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:19.343730 2641 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: TxnStatusTablet state changed. Reason: New leader ec40c6729e344df49eec9e089ba5c79b. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.344007 2641 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:19.346680 2639 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.346961 2639 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:19.347361 2665 mvcc.cc:205] Tried to move back new op lower bound from 7334027015483908096 to 7334027015090749440. Current Snapshot: MvccSnapshot[applied={T|T < 7334027015483908096}]
I20260927 18:32:19.348973 2641 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.349195 2641 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P e204f78cb000478dbfd18f7f65ff71f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:19.350597 2658 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.350840 2658 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P ec40c6729e344df49eec9e089ba5c79b: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:32:19.362658 2642 tablet_replica.cc:439] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec40c6729e344df49eec9e089ba5c79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } } }
I20260927 18:32:19.362954 2642 tablet_replica.cc:442] T f26ea12b641b4f348ced0f9bb00dca32 P 420f1328e7584dde8b57bbb7a815c005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:32:19.373950 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.368267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.97.67:37831)
0927 18:32:19.368436 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.368444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.368900 (+ 456us) server_negotiation.cc:407] Connection header received
0927 18:32:19.369137 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.369142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.369217 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.369358 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.369831 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.370548 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.371890 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.372294 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.372422 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.372828 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.373344 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.373535 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.373594 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.373671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:32:19.376392 2642 consensus_queue.cc:1035] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.384799 2675 consensus_queue.cc:1035] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.404317 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.398059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.97.67:51251)
0927 18:32:19.398818 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.398838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.398963 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.399331 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.399335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.399343 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:19.399576 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.399584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.401022 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.401025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.402154 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.402166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.403158 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.403163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.403273 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.403324 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.403758 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.403772 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.403832 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.403926 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":372,"threads_started":1}
I20260927 18:32:19.404322 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.398083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51251 (local address 127.0.97.65:42837)
0927 18:32:19.398263 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.398272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.398929 (+ 657us) server_negotiation.cc:407] Connection header received
0927 18:32:19.399106 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.399112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.399204 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.399326 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.399733 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.400876 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.402285 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.402962 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.403116 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.403512 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.403653 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.403755 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.403882 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.403970 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:32:19.450908 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.443225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37831 (local address 127.0.97.65:44253)
0927 18:32:19.444064 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.444120 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.444331 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.444790 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.444794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.444806 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:19.445181 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.445191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.446725 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.446731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.448705 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.448718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.449712 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.449717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.449937 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.450000 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.450417 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.450425 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.450490 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.450594 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":437,"threads_started":1}
I20260927 18:32:19.450899 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.443250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44253 (local address 127.0.97.67:37831)
0927 18:32:19.443433 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.443437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.444200 (+ 763us) server_negotiation.cc:407] Connection header received
0927 18:32:19.444513 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.444519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.444630 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.444772 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.445354 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.446596 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.448849 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.449506 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.449642 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.450182 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.450322 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.450412 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.450527 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.450604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:32:19.491629 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:19.491905 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:19.495939 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Sending DeleteTablet for 3 replicas of tablet e970eb340c064916bcf5c22ff4a77278
I20260927 18:32:19.497103 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet e970eb340c064916bcf5c22ff4a77278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36794
I20260927 18:32:19.497231 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet e970eb340c064916bcf5c22ff4a77278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57470
I20260927 18:32:19.498502 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet e970eb340c064916bcf5c22ff4a77278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51442
I20260927 18:32:19.500363 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:19.501327 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:19.510267 2686 tablet_replica.cc:330] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: stopping tablet replica
I20260927 18:32:19.510730 2686 raft_consensus.cc:2229] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:19.511066 2686 raft_consensus.cc:2258] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:19.511402 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf (DEFAULT_TABLE table=test-table [id=16966508b0c94cffa4d0053325c79990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:19.511762 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf (DEFAULT_TABLE table=test-table [id=16966508b0c94cffa4d0053325c79990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:19.511916 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b70eb3268844d1bbe91a8208e95bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:19.511893 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf (DEFAULT_TABLE table=test-table [id=16966508b0c94cffa4d0053325c79990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:19.512394 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b70eb3268844d1bbe91a8208e95bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:19.512428 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b70eb3268844d1bbe91a8208e95bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:19.520407 2686 ts_tablet_manager.cc:1905] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:19.525888 2688 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Bootstrap starting.
I20260927 18:32:19.528795 2688 tablet_bootstrap.cc:654] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:19.531934 2686 ts_tablet_manager.cc:1918] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:19.532159 2686 log.cc:1196] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/e970eb340c064916bcf5c22ff4a77278
I20260927 18:32:19.532817 2686 ts_tablet_manager.cc:1939] T e970eb340c064916bcf5c22ff4a77278 P ec40c6729e344df49eec9e089ba5c79b: Deleting consensus metadata
I20260927 18:32:19.537688 2688 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: No bootstrap required, opened a new log
I20260927 18:32:19.538019 2688 ts_tablet_manager.cc:1397] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 18:32:19.538319 2690 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Bootstrap starting.
I20260927 18:32:19.539021 2688 raft_consensus.cc:348] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.539390 2688 raft_consensus.cc:374] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:19.539520 2688 raft_consensus.cc:729] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Initialized, Role: FOLLOWER
I20260927 18:32:19.539844 2688 consensus_queue.cc:260] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.540323 2690 tablet_bootstrap.cc:654] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:19.542590 2685 tablet_replica.cc:330] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: stopping tablet replica
I20260927 18:32:19.543078 2685 raft_consensus.cc:2229] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:19.543247 2688 ts_tablet_manager.cc:1428] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:32:19.543465 2685 raft_consensus.cc:2258] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:19.544344 2687 tablet_replica.cc:330] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: stopping tablet replica
I20260927 18:32:19.544777 2687 raft_consensus.cc:2229] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:19.545565 2689 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Bootstrap starting.
I20260927 18:32:19.547670 2689 tablet_bootstrap.cc:654] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:19.547737 2685 ts_tablet_manager.cc:1905] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:19.549358 2690 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: No bootstrap required, opened a new log
I20260927 18:32:19.549513 2690 ts_tablet_manager.cc:1397] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260927 18:32:19.549976 2115 catalog_manager.cc:4841] TS ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837): tablet e970eb340c064916bcf5c22ff4a77278 (table test-table [id=2f631d91d0ef46de958677672c11c006]) successfully deleted
I20260927 18:32:19.550230 2687 raft_consensus.cc:2258] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:19.550462 2690 raft_consensus.cc:348] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.550712 2690 raft_consensus.cc:374] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:19.550832 2690 raft_consensus.cc:729] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Initialized, Role: FOLLOWER
I20260927 18:32:19.551090 2690 consensus_queue.cc:260] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.551649 2690 ts_tablet_manager.cc:1428] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:32:19.552434 2687 ts_tablet_manager.cc:1905] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:19.557967 2689 tablet_bootstrap.cc:492] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: No bootstrap required, opened a new log
I20260927 18:32:19.558162 2689 ts_tablet_manager.cc:1397] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260927 18:32:19.559015 2689 raft_consensus.cc:348] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.559311 2689 raft_consensus.cc:374] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:19.559417 2689 raft_consensus.cc:729] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Initialized, Role: FOLLOWER
I20260927 18:32:19.559788 2689 consensus_queue.cc:260] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.560652 2689 ts_tablet_manager.cc:1428] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:19.569103 2641 raft_consensus.cc:482] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:19.569339 2641 raft_consensus.cc:504] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.570117 2685 ts_tablet_manager.cc:1918] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:19.570284 2685 log.cc:1196] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/e970eb340c064916bcf5c22ff4a77278
I20260927 18:32:19.570832 2685 ts_tablet_manager.cc:1939] T e970eb340c064916bcf5c22ff4a77278 P e204f78cb000478dbfd18f7f65ff71f2: Deleting consensus metadata
I20260927 18:32:19.571141 2641 leader_election.cc:288] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831)
I20260927 18:32:19.574172 2687 ts_tablet_manager.cc:1918] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:19.574450 2687 log.cc:1196] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/e970eb340c064916bcf5c22ff4a77278
I20260927 18:32:19.576460 2687 ts_tablet_manager.cc:1939] T e970eb340c064916bcf5c22ff4a77278 P 420f1328e7584dde8b57bbb7a815c005: Deleting consensus metadata
I20260927 18:32:19.578408 2115 catalog_manager.cc:4841] TS 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831): tablet e970eb340c064916bcf5c22ff4a77278 (table test-table [id=2f631d91d0ef46de958677672c11c006]) successfully deleted
I20260927 18:32:19.579908 2116 catalog_manager.cc:4841] TS e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597): tablet e970eb340c064916bcf5c22ff4a77278 (table test-table [id=2f631d91d0ef46de958677672c11c006]) successfully deleted
I20260927 18:32:19.604406 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.571770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34703 (local address 127.0.97.65:42837)
0927 18:32:19.571973 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.571980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.585289 (+ 13309us) server_negotiation.cc:407] Connection header received
0927 18:32:19.585510 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.585528 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.585614 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.585732 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:19.593282 (+ 7550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.594549 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.597795 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.599314 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.599432 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.599566 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.603894 (+ 4328us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.604066 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:32:19.605316 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.571383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.97.66:34703)
0927 18:32:19.585167 (+ 13784us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.585192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.585366 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.592554 (+ 7188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.592560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.592598 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:19.593042 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.593058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.594769 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.594775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.596887 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.596904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.599812 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.599819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.600019 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.600060 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.604705 (+ 4645us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.604928 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13621,"thread_start_us":14094,"threads_started":1}
I20260927 18:32:19.610611 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b70eb3268844d1bbe91a8208e95bdf" candidate_uuid: "e204f78cb000478dbfd18f7f65ff71f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b" is_pre_election: true
I20260927 18:32:19.610955 2287 raft_consensus.cc:2454] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e204f78cb000478dbfd18f7f65ff71f2 in term 0.
I20260927 18:32:19.611624 2359 leader_election.cc:302] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e204f78cb000478dbfd18f7f65ff71f2, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:19.612075 2641 raft_consensus.cc:2781] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:19.612210 2641 raft_consensus.cc:482] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:19.612313 2641 raft_consensus.cc:3037] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.615489 2641 raft_consensus.cc:504] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.616240 2641 leader_election.cc:288] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [CANDIDATE]: Term 1 election: Requested vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831)
I20260927 18:32:19.616986 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b70eb3268844d1bbe91a8208e95bdf" candidate_uuid: "e204f78cb000478dbfd18f7f65ff71f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b"
I20260927 18:32:19.617247 2287 raft_consensus.cc:3037] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.620627 2287 raft_consensus.cc:2454] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e204f78cb000478dbfd18f7f65ff71f2 in term 1.
I20260927 18:32:19.621333 2359 leader_election.cc:302] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e204f78cb000478dbfd18f7f65ff71f2, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:19.621665 2641 raft_consensus.cc:2781] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:19.629868 2641 raft_consensus.cc:686] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 1 LEADER]: Becoming Leader. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Running, Role: LEADER
I20260927 18:32:19.630329 2641 consensus_queue.cc:237] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:19.635238 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.586028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37485 (local address 127.0.97.67:37831)
0927 18:32:19.586219 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.586226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.605709 (+ 19483us) server_negotiation.cc:407] Connection header received
0927 18:32:19.606118 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.606125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.606215 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.606304 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:19.607011 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.608145 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.632102 (+ 23957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.633738 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.633891 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.634027 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.634762 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.634912 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:32:19.636020 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.586405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37831 (local address 127.0.97.66:37485)
0927 18:32:19.605603 (+ 19198us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.605627 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.605893 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.606534 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.606539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.606567 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:19.606736 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.606747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.608347 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.608353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.623823 (+ 15470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.623853 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.634283 (+ 10430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.634290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.634527 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.634574 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.635502 (+ 928us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.635711 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19067,"thread_start_us":23423,"threads_started":1}
I20260927 18:32:19.636739 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b70eb3268844d1bbe91a8208e95bdf" candidate_uuid: "e204f78cb000478dbfd18f7f65ff71f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420f1328e7584dde8b57bbb7a815c005" is_pre_election: true
I20260927 18:32:19.637018 2570 raft_consensus.cc:2454] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e204f78cb000478dbfd18f7f65ff71f2 in term 0.
I20260927 18:32:19.637759 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b70eb3268844d1bbe91a8208e95bdf" candidate_uuid: "e204f78cb000478dbfd18f7f65ff71f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420f1328e7584dde8b57bbb7a815c005"
I20260927 18:32:19.638010 2570 raft_consensus.cc:3037] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:19.641654 2570 raft_consensus.cc:2454] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e204f78cb000478dbfd18f7f65ff71f2 in term 1.
I20260927 18:32:19.645188 2129 catalog_manager.cc:5495] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "e204f78cb000478dbfd18f7f65ff71f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:19.671239 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.664001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.0.97.66:44597)
0927 18:32:19.664161 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.664167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.664449 (+ 282us) server_negotiation.cc:407] Connection header received
0927 18:32:19.664654 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.664660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.664769 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.664947 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.665332 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.666500 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.668544 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.669238 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.669362 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.669873 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.670590 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.670742 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.670866 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.670950 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:32:19.690909 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.684098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50473 (local address 127.0.97.65:42837)
0927 18:32:19.684317 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.684324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.684449 (+ 125us) server_negotiation.cc:407] Connection header received
0927 18:32:19.684732 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.684739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.684839 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.684940 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.685478 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.686731 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.688762 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.689460 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.689592 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.690142 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.690289 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.690385 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.690537 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.690627 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:32:19.691044 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.683738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42837 (local address 127.0.97.66:50473)
0927 18:32:19.684349 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.684370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.684561 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.684965 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.684971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.684983 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:19.685299 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.685310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.686874 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.686881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.688599 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.688614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.689656 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.689663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.689869 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.689938 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.690399 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.690411 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.690496 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.690637 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":596,"threads_started":1}
I20260927 18:32:19.702427 2287 raft_consensus.cc:1261] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Refusing update from remote peer e204f78cb000478dbfd18f7f65ff71f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:19.702489 2570 raft_consensus.cc:1261] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Refusing update from remote peer e204f78cb000478dbfd18f7f65ff71f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:19.703145 2641 consensus_queue.cc:1035] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.703429 2696 consensus_queue.cc:1035] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:19.720477 2663 mvcc.cc:205] Tried to move back new op lower bound from 7334027017015074816 to 7334027016730664960. Current Snapshot: MvccSnapshot[applied={T|T < 7334027017015074816}]
I20260927 18:32:19.720806 2664 mvcc.cc:205] Tried to move back new op lower bound from 7334027017015074816 to 7334027016730664960. Current Snapshot: MvccSnapshot[applied={T|T < 7334027017015074816}]
I20260927 18:32:19.746721 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.739811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44597 (local address 127.0.97.65:48987)
0927 18:32:19.740176 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:19.740197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:19.740310 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:19.740826 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:19.740831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:19.740848 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:32:19.741055 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.741066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.742558 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.742565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.744183 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:19.744197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.745344 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.745350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:19.745551 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.745599 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.746112 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.746124 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:32:19.746197 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:32:19.746386 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260927 18:32:19.747260 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:19.740370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48987 (local address 127.0.97.66:44597)
0927 18:32:19.740529 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:19.740537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:19.740557 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:32:19.740619 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:19.740625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:19.740701 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:19.740795 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:19.741222 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.742390 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.744338 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:19.745072 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:19.745207 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:19.745854 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:19.745986 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:19.746065 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:19.746905 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 18:32:19.746994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:32:19.748464 2665 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:32:21.376327 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:21.376534 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:21.380596 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Sending DeleteTablet for 3 replicas of tablet 22b70eb3268844d1bbe91a8208e95bdf
I20260927 18:32:21.381381 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36794
I20260927 18:32:21.381712 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51442
I20260927 18:32:21.381717 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b70eb3268844d1bbe91a8208e95bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57470
I20260927 18:32:21.382112 2728 tablet_replica.cc:330] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: stopping tablet replica
I20260927 18:32:21.382535 2729 tablet_replica.cc:330] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: stopping tablet replica
I20260927 18:32:21.382839 2729 raft_consensus.cc:2229] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:21.383493 2729 raft_consensus.cc:2258] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:21.384153 2730 tablet_replica.cc:330] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: stopping tablet replica
I20260927 18:32:21.384368 2730 raft_consensus.cc:2229] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:21.384616 2730 raft_consensus.cc:2258] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:21.385439 2730 ts_tablet_manager.cc:1905] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.386488 2729 ts_tablet_manager.cc:1905] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.392385 2730 ts_tablet_manager.cc:1918] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:21.392517 2730 log.cc:1196] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/22b70eb3268844d1bbe91a8208e95bdf
I20260927 18:32:21.392891 2728 raft_consensus.cc:2229] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:21.392988 2730 ts_tablet_manager.cc:1939] T 22b70eb3268844d1bbe91a8208e95bdf P 420f1328e7584dde8b57bbb7a815c005: Deleting consensus metadata
I20260927 18:32:21.393143 2728 raft_consensus.cc:2258] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:21.393154 2729 ts_tablet_manager.cc:1918] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:21.393705 2729 log.cc:1196] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/22b70eb3268844d1bbe91a8208e95bdf
I20260927 18:32:21.394119 2728 ts_tablet_manager.cc:1905] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.394290 2729 ts_tablet_manager.cc:1939] T 22b70eb3268844d1bbe91a8208e95bdf P e204f78cb000478dbfd18f7f65ff71f2: Deleting consensus metadata
I20260927 18:32:21.394276 2115 catalog_manager.cc:4841] TS 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831): tablet 22b70eb3268844d1bbe91a8208e95bdf (table test-table [id=16966508b0c94cffa4d0053325c79990]) successfully deleted
I20260927 18:32:21.395233 2116 catalog_manager.cc:4841] TS e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597): tablet 22b70eb3268844d1bbe91a8208e95bdf (table test-table [id=16966508b0c94cffa4d0053325c79990]) successfully deleted
I20260927 18:32:21.401618 2728 ts_tablet_manager.cc:1918] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:21.401760 2728 log.cc:1196] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/22b70eb3268844d1bbe91a8208e95bdf
I20260927 18:32:21.402172 2728 ts_tablet_manager.cc:1939] T 22b70eb3268844d1bbe91a8208e95bdf P ec40c6729e344df49eec9e089ba5c79b: Deleting consensus metadata
I20260927 18:32:21.410004 2115 catalog_manager.cc:4841] TS ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837): tablet 22b70eb3268844d1bbe91a8208e95bdf (table test-table [id=16966508b0c94cffa4d0053325c79990]) successfully deleted
Sep 27 18:32:21 dist-test-slave-n3p4 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:32:21 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:21.447264 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:21.430998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.0.97.126:39667)
0927 18:32:21.431838 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:21.431843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:21.431866 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:21.431923 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:21.431927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:21.431989 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:21.432147 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:21.433614 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:21.435083 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:21.436379 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:21.436817 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:21.436976 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:21.441921 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:21.441926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:21.441930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:21.441963 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:21.444511 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:21.445365 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:21.445369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:21.445372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:21.445499 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:21.445939 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:21.445942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:21.445944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:21.446216 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:21.446495 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:21.446748 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:32:21.446941 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":552,"threads_started":1}
I20260927 18:32:21.450707 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:21.451323 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:21.457643 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b876a16304e23b828c1198d364899 (DEFAULT_TABLE table=test-table [id=888248251e184b37acc7c8cb00029838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.458105 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b876a16304e23b828c1198d364899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.458016 2549 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b876a16304e23b828c1198d364899 (DEFAULT_TABLE table=test-table [id=888248251e184b37acc7c8cb00029838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.458094 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b876a16304e23b828c1198d364899 (DEFAULT_TABLE table=test-table [id=888248251e184b37acc7c8cb00029838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.458349 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b876a16304e23b828c1198d364899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.458485 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b876a16304e23b828c1198d364899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.463559 2739 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Bootstrap starting.
I20260927 18:32:21.465207 2740 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Bootstrap starting.
I20260927 18:32:21.465562 2741 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Bootstrap starting.
I20260927 18:32:21.466590 2739 tablet_bootstrap.cc:654] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.466770 2740 tablet_bootstrap.cc:654] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.466922 2741 tablet_bootstrap.cc:654] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.468693 2739 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: No bootstrap required, opened a new log
I20260927 18:32:21.468792 2740 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: No bootstrap required, opened a new log
I20260927 18:32:21.468827 2739 ts_tablet_manager.cc:1397] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:32:21.468917 2740 ts_tablet_manager.cc:1397] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:32:21.469530 2741 tablet_bootstrap.cc:492] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: No bootstrap required, opened a new log
I20260927 18:32:21.469656 2741 ts_tablet_manager.cc:1397] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:32:21.469591 2739 raft_consensus.cc:348] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.469594 2740 raft_consensus.cc:348] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.469838 2740 raft_consensus.cc:374] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.469838 2739 raft_consensus.cc:374] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.469934 2740 raft_consensus.cc:729] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.469935 2739 raft_consensus.cc:729] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.470160 2739 consensus_queue.cc:260] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.470160 2740 consensus_queue.cc:260] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.470369 2741 raft_consensus.cc:348] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.470629 2741 raft_consensus.cc:374] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.470733 2741 raft_consensus.cc:729] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.470750 2740 ts_tablet_manager.cc:1428] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:21.470988 2739 ts_tablet_manager.cc:1428] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:32:21.470988 2741 consensus_queue.cc:260] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.471520 2741 ts_tablet_manager.cc:1428] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:21.634698 2745 raft_consensus.cc:482] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:21.634927 2745 raft_consensus.cc:504] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.635603 2745 leader_election.cc:288] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:21.636227 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b876a16304e23b828c1198d364899" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b" is_pre_election: true
I20260927 18:32:21.636209 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b876a16304e23b828c1198d364899" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2" is_pre_election: true
I20260927 18:32:21.636471 2286 raft_consensus.cc:2454] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
I20260927 18:32:21.636492 2426 raft_consensus.cc:2454] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
I20260927 18:32:21.636942 2498 leader_election.cc:302] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:21.637202 2745 raft_consensus.cc:2781] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:21.637300 2745 raft_consensus.cc:482] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:21.637409 2745 raft_consensus.cc:3037] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.639434 2745 raft_consensus.cc:504] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.639958 2745 leader_election.cc:288] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Requested vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:21.640432 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b876a16304e23b828c1198d364899" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2"
I20260927 18:32:21.640470 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b876a16304e23b828c1198d364899" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b"
I20260927 18:32:21.640645 2426 raft_consensus.cc:3037] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.640698 2286 raft_consensus.cc:3037] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.642683 2286 raft_consensus.cc:2454] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:21.643114 2426 raft_consensus.cc:2454] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:21.643133 2498 leader_election.cc:302] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:21.643409 2745 raft_consensus.cc:2781] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:21.643800 2745 raft_consensus.cc:686] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Becoming Leader. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Running, Role: LEADER
I20260927 18:32:21.644074 2745 consensus_queue.cc:237] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.646921 2128 catalog_manager.cc:5495] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "420f1328e7584dde8b57bbb7a815c005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } health_report { overall_health: HEALTHY } } }
I20260927 18:32:21.667688 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:21.662709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.0.97.67:37831)
0927 18:32:21.663266 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:21.663270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:21.663287 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:32:21.663332 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:21.663335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:21.663390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:21.663495 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:21.663884 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:21.664620 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:21.665990 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:21.666402 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:21.666535 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:21.666955 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:21.667055 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:21.667160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:21.667298 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:32:21.667369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":287,"threads_started":1}
I20260927 18:32:21.675830 2426 raft_consensus.cc:1261] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Refusing update from remote peer 420f1328e7584dde8b57bbb7a815c005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:21.675895 2286 raft_consensus.cc:1261] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Refusing update from remote peer 420f1328e7584dde8b57bbb7a815c005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:21.676712 2746 consensus_queue.cc:1035] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:21.677043 2745 consensus_queue.cc:1035] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:21.719359 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:21.719623 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:21.722385 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Sending DeleteTablet for 3 replicas of tablet 3b7b876a16304e23b828c1198d364899
I20260927 18:32:21.723351 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b876a16304e23b828c1198d364899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57470
I20260927 18:32:21.723397 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b876a16304e23b828c1198d364899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36794
I20260927 18:32:21.723671 2729 tablet_replica.cc:330] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: stopping tablet replica
I20260927 18:32:21.723673 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b876a16304e23b828c1198d364899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:21 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51442
I20260927 18:32:21.723878 2729 raft_consensus.cc:2229] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:21.723958 2730 tablet_replica.cc:330] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: stopping tablet replica
I20260927 18:32:21.724102 2729 raft_consensus.cc:2258] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:21.724154 2730 raft_consensus.cc:2229] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:21.724385 2728 tablet_replica.cc:330] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: stopping tablet replica
I20260927 18:32:21.724442 2730 raft_consensus.cc:2258] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:21.724591 2728 raft_consensus.cc:2229] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:21.724449 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:32:21.724799 2728 raft_consensus.cc:2258] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 18:32:21.725116 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:21.725616 2730 ts_tablet_manager.cc:1905] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.726137 2729 ts_tablet_manager.cc:1905] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.726186 2728 ts_tablet_manager.cc:1905] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:21.731290 2728 ts_tablet_manager.cc:1918] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:21.731453 2728 log.cc:1196] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/3b7b876a16304e23b828c1198d364899
I20260927 18:32:21.731815 2728 ts_tablet_manager.cc:1939] T 3b7b876a16304e23b828c1198d364899 P ec40c6729e344df49eec9e089ba5c79b: Deleting consensus metadata
I20260927 18:32:21.732549 2730 ts_tablet_manager.cc:1918] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:21.732707 2730 log.cc:1196] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/3b7b876a16304e23b828c1198d364899
I20260927 18:32:21.732780 2115 catalog_manager.cc:4841] TS ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837): tablet 3b7b876a16304e23b828c1198d364899 (table test-table [id=888248251e184b37acc7c8cb00029838]) successfully deleted
I20260927 18:32:21.732940 2729 ts_tablet_manager.cc:1918] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:32:21.733096 2729 log.cc:1196] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/3b7b876a16304e23b828c1198d364899
I20260927 18:32:21.733129 2730 ts_tablet_manager.cc:1939] T 3b7b876a16304e23b828c1198d364899 P 420f1328e7584dde8b57bbb7a815c005: Deleting consensus metadata
I20260927 18:32:21.733513 2729 ts_tablet_manager.cc:1939] T 3b7b876a16304e23b828c1198d364899 P e204f78cb000478dbfd18f7f65ff71f2: Deleting consensus metadata
I20260927 18:32:21.734170 2115 catalog_manager.cc:4841] TS 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831): tablet 3b7b876a16304e23b828c1198d364899 (table test-table [id=888248251e184b37acc7c8cb00029838]) successfully deleted
I20260927 18:32:21.734880 2116 catalog_manager.cc:4841] TS e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597): tablet 3b7b876a16304e23b828c1198d364899 (table test-table [id=888248251e184b37acc7c8cb00029838]) successfully deleted
I20260927 18:32:21.738742 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 (DEFAULT_TABLE table=test-table [id=67841f34c78641b2839fa81d4eb59f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.738816 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 (DEFAULT_TABLE table=test-table [id=67841f34c78641b2839fa81d4eb59f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.739166 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512d11c4f88740eb8a4c8f8db4b327b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.739189 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512d11c4f88740eb8a4c8f8db4b327b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.739102 2549 tablet_service.cc:1467] Processing CreateTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 (DEFAULT_TABLE table=test-table [id=67841f34c78641b2839fa81d4eb59f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:21.739476 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512d11c4f88740eb8a4c8f8db4b327b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:21.743348 2740 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Bootstrap starting.
I20260927 18:32:21.744608 2740 tablet_bootstrap.cc:654] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.745350 2741 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Bootstrap starting.
I20260927 18:32:21.746222 2739 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Bootstrap starting.
I20260927 18:32:21.746922 2741 tablet_bootstrap.cc:654] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.747180 2740 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: No bootstrap required, opened a new log
I20260927 18:32:21.747408 2740 ts_tablet_manager.cc:1397] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260927 18:32:21.748202 2739 tablet_bootstrap.cc:654] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:21.748718 2740 raft_consensus.cc:348] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.748951 2740 raft_consensus.cc:374] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.749040 2740 raft_consensus.cc:729] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec40c6729e344df49eec9e089ba5c79b, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.749328 2740 consensus_queue.cc:260] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.749827 2741 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: No bootstrap required, opened a new log
I20260927 18:32:21.749975 2741 ts_tablet_manager.cc:1397] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:21.749984 2740 ts_tablet_manager.cc:1428] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:32:21.750545 2741 raft_consensus.cc:348] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.750785 2741 raft_consensus.cc:374] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.750869 2741 raft_consensus.cc:729] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e204f78cb000478dbfd18f7f65ff71f2, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.751091 2741 consensus_queue.cc:260] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.751621 2741 ts_tablet_manager.cc:1428] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:21.752851 2739 tablet_bootstrap.cc:492] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: No bootstrap required, opened a new log
I20260927 18:32:21.753032 2739 ts_tablet_manager.cc:1397] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 18:32:21.753768 2739 raft_consensus.cc:348] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.754079 2739 raft_consensus.cc:374] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:21.754166 2739 raft_consensus.cc:729] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Initialized, Role: FOLLOWER
I20260927 18:32:21.754386 2739 consensus_queue.cc:260] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.754906 2739 ts_tablet_manager.cc:1428] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:32:21.797761 2746 raft_consensus.cc:482] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:21.798084 2746 raft_consensus.cc:504] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.799196 2746 leader_election.cc:288] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:21.799340 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512d11c4f88740eb8a4c8f8db4b327b9" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2" is_pre_election: true
I20260927 18:32:21.799640 2426 raft_consensus.cc:2454] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
I20260927 18:32:21.799680 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512d11c4f88740eb8a4c8f8db4b327b9" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b" is_pre_election: true
I20260927 18:32:21.799937 2286 raft_consensus.cc:2454] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 0.
I20260927 18:32:21.800091 2499 leader_election.cc:302] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, e204f78cb000478dbfd18f7f65ff71f2; no voters:
I20260927 18:32:21.800418 2746 raft_consensus.cc:2781] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:21.800566 2746 raft_consensus.cc:482] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:21.800707 2746 raft_consensus.cc:3037] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.802848 2746 raft_consensus.cc:504] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.803449 2746 leader_election.cc:288] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Requested vote from peers ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837), e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597)
I20260927 18:32:21.803933 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512d11c4f88740eb8a4c8f8db4b327b9" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e204f78cb000478dbfd18f7f65ff71f2"
I20260927 18:32:21.803968 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512d11c4f88740eb8a4c8f8db4b327b9" candidate_uuid: "420f1328e7584dde8b57bbb7a815c005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec40c6729e344df49eec9e089ba5c79b"
I20260927 18:32:21.804183 2286 raft_consensus.cc:3037] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.804183 2426 raft_consensus.cc:3037] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:21.806322 2286 raft_consensus.cc:2454] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:21.806594 2426 raft_consensus.cc:2454] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420f1328e7584dde8b57bbb7a815c005 in term 1.
I20260927 18:32:21.806710 2498 leader_election.cc:302] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420f1328e7584dde8b57bbb7a815c005, ec40c6729e344df49eec9e089ba5c79b; no voters:
I20260927 18:32:21.806983 2746 raft_consensus.cc:2781] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:21.807168 2746 raft_consensus.cc:686] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Becoming Leader. State: Replica: 420f1328e7584dde8b57bbb7a815c005, State: Running, Role: LEADER
I20260927 18:32:21.807415 2746 consensus_queue.cc:237] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } }
I20260927 18:32:21.810048 2127 catalog_manager.cc:5495] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "420f1328e7584dde8b57bbb7a815c005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420f1328e7584dde8b57bbb7a815c005" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37831 } health_report { overall_health: HEALTHY } } }
I20260927 18:32:21.843659 2426 raft_consensus.cc:1261] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Refusing update from remote peer 420f1328e7584dde8b57bbb7a815c005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:21.844159 2286 raft_consensus.cc:1261] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Refusing update from remote peer 420f1328e7584dde8b57bbb7a815c005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:21.844180 2745 consensus_queue.cc:1035] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e204f78cb000478dbfd18f7f65ff71f2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:21.844735 2746 consensus_queue.cc:1035] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec40c6729e344df49eec9e089ba5c79b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:32:21.863489 2709 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:32:23.442794 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:23.443056 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:23.446889 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254ca1cb80344f9dbad22c8d45f63972: Sending DeleteTablet for 3 replicas of tablet 512d11c4f88740eb8a4c8f8db4b327b9
I20260927 18:32:23.447690 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:23 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36794
I20260927 18:32:23.447780 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:23 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51442
I20260927 18:32:23.447762 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 512d11c4f88740eb8a4c8f8db4b327b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:23 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57470
I20260927 18:32:23.448396 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2201
I20260927 18:32:23.461275 2781 tablet_replica.cc:330] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: stopping tablet replica
I20260927 18:32:23.461686 2781 raft_consensus.cc:2229] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:23.462114 2781 raft_consensus.cc:2258] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:23.462759 2782 tablet_replica.cc:330] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: stopping tablet replica
I20260927 18:32:23.463609 2783 tablet_replica.cc:330] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: stopping tablet replica
I20260927 18:32:23.463815 2783 raft_consensus.cc:2229] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:23.463966 2781 ts_tablet_manager.cc:1905] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:23.464049 2783 raft_consensus.cc:2258] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:23.464551 2782 raft_consensus.cc:2229] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:23.464994 2782 raft_consensus.cc:2258] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:23.465019 2783 ts_tablet_manager.cc:1905] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:23.466409 2782 ts_tablet_manager.cc:1905] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:23.470760 2781 ts_tablet_manager.cc:1918] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:23.470975 2781 log.cc:1196] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/512d11c4f88740eb8a4c8f8db4b327b9
I20260927 18:32:23.471266 2783 ts_tablet_manager.cc:1918] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:23.471398 2783 log.cc:1196] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/512d11c4f88740eb8a4c8f8db4b327b9
I20260927 18:32:23.471494 2781 ts_tablet_manager.cc:1939] T 512d11c4f88740eb8a4c8f8db4b327b9 P ec40c6729e344df49eec9e089ba5c79b: Deleting consensus metadata
I20260927 18:32:23.471843 2783 ts_tablet_manager.cc:1939] T 512d11c4f88740eb8a4c8f8db4b327b9 P e204f78cb000478dbfd18f7f65ff71f2: Deleting consensus metadata
I20260927 18:32:23.472623 2115 catalog_manager.cc:4841] TS ec40c6729e344df49eec9e089ba5c79b (127.0.97.65:42837): tablet 512d11c4f88740eb8a4c8f8db4b327b9 (table test-table [id=67841f34c78641b2839fa81d4eb59f74]) successfully deleted
I20260927 18:32:23.472733 2782 ts_tablet_manager.cc:1918] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:32:23.472796 2116 catalog_manager.cc:4841] TS e204f78cb000478dbfd18f7f65ff71f2 (127.0.97.66:44597): tablet 512d11c4f88740eb8a4c8f8db4b327b9 (table test-table [id=67841f34c78641b2839fa81d4eb59f74]) successfully deleted
I20260927 18:32:23.472837 2782 log.cc:1196] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/512d11c4f88740eb8a4c8f8db4b327b9
I20260927 18:32:23.473219 2782 ts_tablet_manager.cc:1939] T 512d11c4f88740eb8a4c8f8db4b327b9 P 420f1328e7584dde8b57bbb7a815c005: Deleting consensus metadata
I20260927 18:32:23.474105 2115 catalog_manager.cc:4841] TS 420f1328e7584dde8b57bbb7a815c005 (127.0.97.67:37831): tablet 512d11c4f88740eb8a4c8f8db4b327b9 (table test-table [id=67841f34c78641b2839fa81d4eb59f74]) successfully deleted
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:23.483958 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:23.451272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.0.97.65:42837)
0927 18:32:23.463791 (+ 12519us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:23.463795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:23.463816 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:23.463867 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:23.463871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:23.463983 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:23.464191 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:23.465919 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.466993 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.468777 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.469444 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.469592 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:23.478070 (+ 8478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:23.478085 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:23.478102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:23.478146 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:23.480639 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.481346 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.481353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.481355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.481936 (+ 581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.482393 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.482397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.482399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.482757 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:23.483074 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:23.483377 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:32:23.483653 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12356,"thread_start_us":11892,"threads_started":1}
W20260927 18:32:23.484558 2332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36808
W20260927 18:32:23.485179 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:23.485401 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2201
I20260927 18:32:23.507485 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2341
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:23.522212 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:23.508215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.97.66:44597)
0927 18:32:23.508747 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:23.508751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:23.508767 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:32:23.508819 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:23.508822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:23.508878 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:23.508964 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:23.510265 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.511104 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.512382 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.512773 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.512866 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:23.516924 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:23.516939 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:23.516955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:23.516995 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:23.519806 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.520401 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.520408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.520410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.520544 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.520851 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.520854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.520856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.521184 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:23.521384 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:23.521638 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:32:23.521877 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
W20260927 18:32:23.522886 2471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484
W20260927 18:32:23.523485 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:23.523636 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2341
I20260927 18:32:23.545583 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2480
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:23.560341 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:23.546379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.0.97.67:37831)
0927 18:32:23.546954 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:23.546959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:23.546976 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:32:23.547041 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:23.547045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:23.547112 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:23.547207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:23.548423 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.549194 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.550467 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:23.550869 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:23.550968 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:23.555077 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:23.555091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:23.555108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:23.555138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:23.557727 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.558485 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.558492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.558495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.558653 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:23.558969 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:23.558972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:23.558974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:23.559302 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:23.559507 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:23.559852 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:32:23.560078 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":294,"threads_started":1}
W20260927 18:32:23.561033 2610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51476
W20260927 18:32:23.561623 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:23.561755 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2480
I20260927 18:32:23.583498 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2097
I20260927 18:32:23.584717 2169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:23.701879 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2097
2026-09-27T18:32:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8436 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2791](info): set up 2 sockets
Sep 27 18:32:23 dist-test-slave-n3p4 krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260927 18:32:24.794359 389 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:24 dist-test-slave-n3p4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:24.982213 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34841
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:25.300348 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:25.300722 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:25.300894 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:25.308043 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:25.308183 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:25.308290 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:25.308396 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:25.308480 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:25.317915 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:25.319207 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:25.320478 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:25.331884 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:25.331988 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:25.332134 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:25.333209 2804 server_base.cc:1034] running on GCE node
I20260927 18:32:25.334136 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:25.335819 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:25.337073 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790533945337001 us; error 60 us; skew 500 ppm
I20260927 18:32:25.337428 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:25 dist-test-slave-n3p4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:25.340620 2804 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:25.342154 2804 webserver.cc:460] Webserver started at http://127.0.97.126:35593/ using document root <none> and password file <none>
I20260927 18:32:25.342929 2804 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:25.343071 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:25.343384 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:25.346455 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "20e3563ea659437697d5c77c13aa5598"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "cf6edcbde395aa2b636066530ea131cc"
server_key_iv: "ed4920f79c1b4e5af8581080123beaf1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.347291 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "20e3563ea659437697d5c77c13aa5598"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "cf6edcbde395aa2b636066530ea131cc"
server_key_iv: "ed4920f79c1b4e5af8581080123beaf1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.353015 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:25.358569 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:25.360378 2804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:32:25.360556 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "20e3563ea659437697d5c77c13aa5598"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "cf6edcbde395aa2b636066530ea131cc"
server_key_iv: "ed4920f79c1b4e5af8581080123beaf1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.360836 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:25.390262 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:25.394284 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:25.394515 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:25.412968 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34841
I20260927 18:32:25.412997 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34841 every 8 connection(s)
I20260927 18:32:25.414393 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:25.418346 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:25.418774 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2804
I20260927 18:32:25.418962 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:25.419613 389 external_mini_cluster.cc:1428] Setting key e544f697c9bf8001494a4c79248b1be6
I20260927 18:32:25.432053 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598: Bootstrap starting.
Sep 27 18:32:25 dist-test-slave-n3p4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:25.435725 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:25.436769 2871 log.cc:824] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:25.439694 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598: No bootstrap required, opened a new log
I20260927 18:32:25.442945 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:25.421490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.97.126:34841)
0927 18:32:25.422259 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:25.422270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:25.422310 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:32:25.424256 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:25.424281 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:25.424668 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:25.424999 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:25.426869 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:25.428174 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:25.430029 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:25.430741 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:25.430879 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:25.435360 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:25.435368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:25.435382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:25.435422 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:25.438660 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:25.439349 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:25.439356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:25.439362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:25.439512 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:25.439922 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:25.439927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:25.439930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:25.440655 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:25.440942 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:25.441331 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:32:25.441774 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":375,"threads_started":1}
I20260927 18:32:25.448511 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } }
I20260927 18:32:25.448889 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:25.448997 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e3563ea659437697d5c77c13aa5598, State: Initialized, Role: FOLLOWER
I20260927 18:32:25.449532 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } }
I20260927 18:32:25.449810 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:25.449960 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:25.450127 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:25.452569 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } }
I20260927 18:32:25.452982 2871 leader_election.cc:302] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e3563ea659437697d5c77c13aa5598; no voters:
I20260927 18:32:25.453529 2871 leader_election.cc:288] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:25.453692 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:25.454391 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [term 1 LEADER]: Becoming Leader. State: Replica: 20e3563ea659437697d5c77c13aa5598, State: Running, Role: LEADER
I20260927 18:32:25.455062 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } }
I20260927 18:32:25.455716 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:25.457690 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e3563ea659437697d5c77c13aa5598. Latest consensus state: current_term: 1 leader_uuid: "20e3563ea659437697d5c77c13aa5598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } } }
I20260927 18:32:25.458238 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:25.458098 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e3563ea659437697d5c77c13aa5598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e3563ea659437697d5c77c13aa5598" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34841 } } }
I20260927 18:32:25.458348 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:25.459326 2882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:25.468873 2882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:25.477600 2882 catalog_manager.cc:1347] Generated new cluster ID: fd16ae22129248edba9a508318cf123a
I20260927 18:32:25.477720 2882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:25.490717 2882 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:25.492467 2882 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:25.498776 2882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e3563ea659437697d5c77c13aa5598: Generated new TSK 0
I20260927 18:32:25.499564 2882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:25.549731 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:25.884603 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:25.884953 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:25.885128 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:25.892583 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:25.892745 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:25.892923 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:25.902804 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:25.904112 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:25.905431 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:25.918563 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:25.918597 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:25.918529 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:25.919765 2897 server_base.cc:1034] running on GCE node
I20260927 18:32:25.920552 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:25.921548 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:25.922783 2897 hybrid_clock.cc:648] HybridClock initialized: now 1790533945922710 us; error 96 us; skew 500 ppm
I20260927 18:32:25.923123 2897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:25 dist-test-slave-n3p4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:25.926163 2897 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:25.927564 2897 webserver.cc:460] Webserver started at http://127.0.97.65:35159/ using document root <none> and password file <none>
I20260927 18:32:25.928239 2897 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:25.928380 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:25.928678 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:25.931516 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "fbe5e008bd0a430c9d448d999ee0b9ab"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "2bf3e5928c36335e4b7b3f375b738e30"
server_key_iv: "4a5dd42fe9230ecd0298d5fda1cd45d1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.932267 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "fbe5e008bd0a430c9d448d999ee0b9ab"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "2bf3e5928c36335e4b7b3f375b738e30"
server_key_iv: "4a5dd42fe9230ecd0298d5fda1cd45d1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.937292 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:32:25.940738 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:25.942117 2897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:25.942281 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "fbe5e008bd0a430c9d448d999ee0b9ab"
format_stamp: "Formatted at 2026-09-27 18:32:25 on dist-test-slave-n3p4"
server_key: "2bf3e5928c36335e4b7b3f375b738e30"
server_key_iv: "4a5dd42fe9230ecd0298d5fda1cd45d1"
server_key_version: "encryptionkey@0"
I20260927 18:32:25.942477 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:25.959581 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:25.963645 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:25.963923 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:25.964895 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:25.966391 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:25.966492 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:25.966697 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:25.966770 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.000064 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42779
I20260927 18:32:26.000113 3024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42779 every 8 connection(s)
I20260927 18:32:26.001353 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:26.010030 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 2897
I20260927 18:32:26.010238 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:26.010838 389 external_mini_cluster.cc:1428] Setting key 01d9cfb8a61c19746151151d7159a41a
Sep 27 18:32:26 dist-test-slave-n3p4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:26.022891 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:26.003976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55881 (local address 127.0.97.126:34841)
0927 18:32:26.004349 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:26.004353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:26.005321 (+ 968us) server_negotiation.cc:407] Connection header received
0927 18:32:26.007220 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:26.007225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:26.007313 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:26.007427 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:26.008998 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.009698 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.011407 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.011972 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.012082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:26.015858 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:26.015867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:26.015875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:26.015921 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:26.019515 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:26.020502 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:26.020508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:26.020510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:26.020637 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:26.021128 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:26.021131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:26.021133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:26.021389 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:26.021496 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:26.022459 (+ 963us) server_negotiation.cc:299] Negotiation successful
0927 18:32:26.022629 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":185,"threads_started":1}
I20260927 18:32:26.024544 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:26.004481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34841 (local address 127.0.97.65:55881)
0927 18:32:26.005158 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:26.005203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:26.006947 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:26.007653 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:26.007661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:26.008168 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:26.008835 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:26.008853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.009872 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.009881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:26.011252 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:26.011263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.012200 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.012207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:26.012487 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:26.013359 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:26.013387 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:26.015633 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:26.019657 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:26.019670 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:26.019692 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:26.020362 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:26.020752 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:26.020760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:26.020767 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:26.021014 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:26.021524 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:26.021535 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:26.022064 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 18:32:26.022376 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 18:32:26.022986 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260927 18:32:26.026227 3025 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34841
I20260927 18:32:26.026578 3025 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:26.027350 3025 heartbeater.cc:511] Master 127.0.97.126:34841 requested a full tablet report, sending...
I20260927 18:32:26.029407 2836 ts_manager.cc:194] Registered new tserver with Master: fbe5e008bd0a430c9d448d999ee0b9ab (127.0.97.65:42779)
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:32:26.030988 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:55881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:26.055287 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:26.374729 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:26.375133 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:26.375308 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:26.382594 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:26.382743 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:26.382925 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:26.392751 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:26.394105 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:26.395395 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:26.409399 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:26.409458 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:26.409343 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:26.410844 3031 server_base.cc:1034] running on GCE node
I20260927 18:32:26.411564 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:26.412690 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:26.413933 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790533946413883 us; error 42 us; skew 500 ppm
I20260927 18:32:26.414274 3031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:26 dist-test-slave-n3p4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:26.417840 3031 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:26.419597 3031 webserver.cc:460] Webserver started at http://127.0.97.66:43667/ using document root <none> and password file <none>
I20260927 18:32:26.420478 3031 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:26.420622 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:26.420975 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:26.425223 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "a66fe38290fa45a08129a666f9fbbb75"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "6202b4850ccfdc78d218de4cb3349f40"
server_key_iv: "d5d3d3883874f4836115e02add2ec34d"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.426329 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "a66fe38290fa45a08129a666f9fbbb75"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "6202b4850ccfdc78d218de4cb3349f40"
server_key_iv: "d5d3d3883874f4836115e02add2ec34d"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.433622 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 18:32:26.437726 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.439235 3031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:26.439412 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "a66fe38290fa45a08129a666f9fbbb75"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "6202b4850ccfdc78d218de4cb3349f40"
server_key_iv: "d5d3d3883874f4836115e02add2ec34d"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.439630 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:26.471041 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:26.475682 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:26.475994 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:26.477021 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:26.478451 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:26.478533 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.478737 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:26.478852 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.513037 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39557
I20260927 18:32:26.513090 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39557 every 8 connection(s)
I20260927 18:32:26.514580 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:26.524463 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3031
I20260927 18:32:26.524654 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:26.525151 389 external_mini_cluster.cc:1428] Setting key 48289eaf26e5f652f832f466991eb56a
Sep 27 18:32:26 dist-test-slave-n3p4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:26.535537 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:26.517313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:38345 (local address 127.0.97.126:34841)
0927 18:32:26.517500 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:26.517504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:26.518896 (+ 1392us) server_negotiation.cc:407] Connection header received
0927 18:32:26.520715 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:26.520721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:26.520810 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:26.520920 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:26.522439 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.523140 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.524721 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.525184 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.525322 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:26.529359 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:26.529365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:26.529368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:26.529402 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:26.532235 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:26.533038 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:26.533045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:26.533048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:26.533210 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:26.533765 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:26.533768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:26.533770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:26.534076 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:26.534181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:26.535117 (+ 936us) server_negotiation.cc:299] Negotiation successful
0927 18:32:26.535304 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:26.537410 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:26.517885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34841 (local address 127.0.97.66:38345)
0927 18:32:26.518709 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:26.518760 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:26.520436 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:26.521150 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:26.521158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:26.521660 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:26.522278 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:26.522295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.523301 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.523305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:26.524561 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:26.524577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:26.525413 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:26.525420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:26.525665 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:26.526471 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:26.526495 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:26.529121 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:26.532396 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:26.532404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:26.532419 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:26.532896 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:26.533365 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:26.533373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:26.533379 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:26.533649 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:26.534196 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:26.534208 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:26.534729 (+ 521us) client_negotiation.cc:769] Sending connection context
0927 18:32:26.535036 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 18:32:26.535678 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":358,"threads_started":1}
I20260927 18:32:26.539011 3159 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34841
I20260927 18:32:26.539320 3159 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:26.540026 3159 heartbeater.cc:511] Master 127.0.97.126:34841 requested a full tablet report, sending...
I20260927 18:32:26.541625 2836 ts_manager.cc:194] Registered new tserver with Master: a66fe38290fa45a08129a666f9fbbb75 (127.0.97.66:39557)
I20260927 18:32:26.542934 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:38345
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:26.571003 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:26.892431 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:26.892722 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:26.892923 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:26.900138 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:26.900249 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:26.900430 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:26.910147 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:26.911420 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:26.912706 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:26.926060 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:26.926047 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:26.926060 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:26.926992 3164 server_base.cc:1034] running on GCE node
I20260927 18:32:26.927654 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:26.928591 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:26.929867 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790533946929808 us; error 43 us; skew 500 ppm
I20260927 18:32:26.930182 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:26 dist-test-slave-n3p4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:26.933195 3164 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:26.934651 3164 webserver.cc:460] Webserver started at http://127.0.97.67:43103/ using document root <none> and password file <none>
I20260927 18:32:26.935410 3164 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:26.935551 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:26.935869 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:26.938962 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "736c3e22746440f98da5988e9b06113f"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "b8e36e98b5faeab862352ef1ad8076e1"
server_key_iv: "cd47642e0861db9630226538caf8221a"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.939749 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "736c3e22746440f98da5988e9b06113f"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "b8e36e98b5faeab862352ef1ad8076e1"
server_key_iv: "cd47642e0861db9630226538caf8221a"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.945329 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 18:32:26.949168 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.950691 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:26.950860 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "736c3e22746440f98da5988e9b06113f"
format_stamp: "Formatted at 2026-09-27 18:32:26 on dist-test-slave-n3p4"
server_key: "b8e36e98b5faeab862352ef1ad8076e1"
server_key_iv: "cd47642e0861db9630226538caf8221a"
server_key_version: "encryptionkey@0"
I20260927 18:32:26.951066 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:26.965463 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:26.969456 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:26.969760 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:26.970819 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:26.972101 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:26.972172 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:26.972302 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:26.972364 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:27.005232 3164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37749
I20260927 18:32:27.005259 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37749 every 8 connection(s)
I20260927 18:32:27.006801 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:27.009318 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3164
I20260927 18:32:27.009495 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:27.010057 389 external_mini_cluster.cc:1428] Setting key 92c944b29fd0c092481f04db87aa5ccb
Sep 27 18:32:27 dist-test-slave-n3p4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:27.027086 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:27.009487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51657 (local address 127.0.97.126:34841)
0927 18:32:27.009699 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:27.009704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:27.011072 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 18:32:27.013049 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:27.013054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:27.013170 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:27.013274 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:27.014730 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:27.015494 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:27.017037 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:27.017447 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:27.017592 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:27.020950 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:27.020955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:27.020958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:27.020987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:27.023814 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:27.024626 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:27.024630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:27.024633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:27.024810 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:27.025340 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:27.025351 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:27.025354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:27.025644 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:27.025821 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:27.026507 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 18:32:27.026729 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:32:27.027988 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:27.010064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34841 (local address 127.0.97.67:51657)
0927 18:32:27.010909 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:27.010944 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:27.012814 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:27.013492 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:27.013499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:27.014047 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:27.014576 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:27.014593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:27.015646 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:27.015656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:27.016900 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:27.016911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:27.017674 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:27.017682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:27.017989 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:27.018633 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:27.018654 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:27.020758 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:27.023952 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:27.023958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:27.023992 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:27.024463 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:27.024931 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:27.024935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:27.024937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:27.025220 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:27.025800 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:27.025809 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:27.026148 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 18:32:27.026420 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 18:32:27.026930 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":307,"threads_started":1}
I20260927 18:32:27.029392 3292 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34841
I20260927 18:32:27.029731 3292 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:27.030542 3292 heartbeater.cc:511] Master 127.0.97.126:34841 requested a full tablet report, sending...
I20260927 18:32:27.032019 2836 ts_manager.cc:194] Registered new tserver with Master: 736c3e22746440f98da5988e9b06113f (127.0.97.67:37749)
I20260927 18:32:27.032858 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51657
I20260927 18:32:27.034041 3025 heartbeater.cc:503] Master 127.0.97.126:34841 was elected leader, sending a full tablet report...
I20260927 18:32:27.037717 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:32:27.067435 3027 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:27.064970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.0.97.126:34841)
0927 18:32:27.065153 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:27.065158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:27.065712 (+ 554us) server_negotiation.cc:407] Connection header received
0927 18:32:27.065922 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:27.065926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:27.065989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:27.066083 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:27.067199 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55618: BlockingRecv error: recv got EOF from 127.0.0.1:55618 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:32:27.070721 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2897
W20260927 18:32:27.073448 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2897: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:42779: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:32:27.073702 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2897
I20260927 18:32:27.089201 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3031
W20260927 18:32:27.091873 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3031: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:39557: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:32:27.092017 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3031
I20260927 18:32:27.107297 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3164
W20260927 18:32:27.110239 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3164: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:37749: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:32:27.110409 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3164
I20260927 18:32:27.125723 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 2804
I20260927 18:32:27.126977 2866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:27.228412 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 2804
2026-09-27T18:32:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3523 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:27 dist-test-slave-n3p4 krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:27 dist-test-slave-n3p4 krb5kdc[3309](info): set up 2 sockets
Sep 27 18:32:27 dist-test-slave-n3p4 krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260927 18:32:28.398214 389 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:28 dist-test-slave-n3p4 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:28.584404 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38499
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:28.907184 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:28.907488 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:28.907701 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:28.914949 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:28.915066 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:28.915179 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:28.915278 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:28.915366 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:28.925525 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:28.926957 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:28.928314 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:28.940341 3327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:28.940433 3328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:28.940371 3330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:28.941700 3322 server_base.cc:1034] running on GCE node
I20260927 18:32:28.942523 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:28.943861 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:28.945086 3322 hybrid_clock.cc:648] HybridClock initialized: now 1790533948945039 us; error 41 us; skew 500 ppm
I20260927 18:32:28.945395 3322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:28 dist-test-slave-n3p4 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:28.953871 3322 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:28.955417 3322 webserver.cc:460] Webserver started at http://127.0.97.126:38059/ using document root <none> and password file <none>
I20260927 18:32:28.956146 3322 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:28.956290 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:28.956632 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:28.959836 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "7d29e79216ff4645946f305e17d1f103"
format_stamp: "Formatted at 2026-09-27 18:32:28 on dist-test-slave-n3p4"
server_key: "283c07d1203601c13b62e493893c970d"
server_key_iv: "30969a248b6671224478aa410845c06b"
server_key_version: "encryptionkey@0"
I20260927 18:32:28.960690 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "7d29e79216ff4645946f305e17d1f103"
format_stamp: "Formatted at 2026-09-27 18:32:28 on dist-test-slave-n3p4"
server_key: "283c07d1203601c13b62e493893c970d"
server_key_iv: "30969a248b6671224478aa410845c06b"
server_key_version: "encryptionkey@0"
I20260927 18:32:28.966092 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 18:32:28.969753 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:28.971239 3322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:32:28.971437 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "7d29e79216ff4645946f305e17d1f103"
format_stamp: "Formatted at 2026-09-27 18:32:28 on dist-test-slave-n3p4"
server_key: "283c07d1203601c13b62e493893c970d"
server_key_iv: "30969a248b6671224478aa410845c06b"
server_key_version: "encryptionkey@0"
I20260927 18:32:28.971673 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:29.011986 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:29.016251 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:29.016568 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:29.035965 3322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38499
I20260927 18:32:29.036024 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38499 every 8 connection(s)
I20260927 18:32:29.037424 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:29.041345 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:29.041723 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3322
I20260927 18:32:29.041975 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:29.042593 389 external_mini_cluster.cc:1428] Setting key 02162dfb0a1c2beb1148ceb9a316bd27
I20260927 18:32:29.054181 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103: Bootstrap starting.
Sep 27 18:32:29 dist-test-slave-n3p4 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:29.057332 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:29.058620 3389 log.cc:824] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:29.062002 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103: No bootstrap required, opened a new log
I20260927 18:32:29.065176 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:29.044394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.97.126:38499)
0927 18:32:29.045198 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:29.045209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:29.045249 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:32:29.047273 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:29.047295 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:29.047689 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:29.048014 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:29.049756 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:29.051163 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:29.052454 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:29.053113 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:29.053328 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:29.057250 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:29.057258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:29.057272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:29.057309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:29.060948 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:29.061701 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:29.061709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:29.061717 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:29.061953 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:29.062366 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:29.062372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:29.062380 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:29.062910 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:29.063210 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:29.063723 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 18:32:29.064153 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":381,"threads_started":1}
I20260927 18:32:29.069027 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:32:29.070334 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } }
I20260927 18:32:29.070744 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:29.070848 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d29e79216ff4645946f305e17d1f103, State: Initialized, Role: FOLLOWER
I20260927 18:32:29.071385 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } }
I20260927 18:32:29.071632 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:29.071772 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:29.071949 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:29.074172 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } }
I20260927 18:32:29.074756 3389 leader_election.cc:302] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d29e79216ff4645946f305e17d1f103; no voters:
I20260927 18:32:29.075445 3389 leader_election.cc:288] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:29.075557 3395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:29.078083 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:29.078032 3395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [term 1 LEADER]: Becoming Leader. State: Replica: 7d29e79216ff4645946f305e17d1f103, State: Running, Role: LEADER
I20260927 18:32:29.079514 3395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } }
I20260927 18:32:29.082088 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d29e79216ff4645946f305e17d1f103. Latest consensus state: current_term: 1 leader_uuid: "7d29e79216ff4645946f305e17d1f103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } } }
I20260927 18:32:29.082702 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d29e79216ff4645946f305e17d1f103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29e79216ff4645946f305e17d1f103" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38499 } } }
I20260927 18:32:29.083137 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:29.083889 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:29.086637 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:29.093340 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:29.105394 3413 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:32:29.108592 3401 catalog_manager.cc:1347] Generated new cluster ID: 7d19f98dcb8146119b9f2f46735374f9
I20260927 18:32:29.108819 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:29.117944 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:29.109094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.0.97.126:38499)
0927 18:32:29.109284 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:29.109290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:29.109903 (+ 613us) server_negotiation.cc:407] Connection header received
0927 18:32:29.110124 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:29.110129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:29.110220 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:29.110335 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:29.111503 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:29.112627 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:29.114340 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:29.115033 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:29.115186 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:29.116346 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:29.116352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:29.116357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:29.117117 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:29.117189 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:29.117214 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:29.117307 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:29.117482 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:32:29.117587 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:32:29.132248 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:29.134300 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:29.161175 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d29e79216ff4645946f305e17d1f103: Generated new TSK 0
I20260927 18:32:29.162350 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:32:29.180915 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3322
I20260927 18:32:29.182246 3384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:29.274097 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3322
2026-09-27T18:32:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2040 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:29 dist-test-slave-n3p4 krb5kdc[3423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:29 dist-test-slave-n3p4 krb5kdc[3423](info): set up 2 sockets
Sep 27 18:32:29 dist-test-slave-n3p4 krb5kdc[3423](info): commencing operation
krb5kdc: starting...
W20260927 18:32:30.362251 389 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:30 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:30.549942 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44461
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:30.865543 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:30.865932 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:30.866094 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:30.873044 3436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:30.873191 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:30.873301 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:30.873405 3436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:30.873494 3436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:30.883060 3436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:30.884354 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:30.885722 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:30.897493 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:30.897608 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:30.897635 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:30.898550 3436 server_base.cc:1034] running on GCE node
I20260927 18:32:30.899401 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:30.900885 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:30.902118 3436 hybrid_clock.cc:648] HybridClock initialized: now 1790533950902095 us; error 69 us; skew 500 ppm
I20260927 18:32:30.902450 3436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:30 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:30.905429 3436 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:30.907059 3436 webserver.cc:460] Webserver started at http://127.0.97.126:44571/ using document root <none> and password file <none>
I20260927 18:32:30.907732 3436 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:30.907868 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:30.908175 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:30.911080 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "1ec1a10b8c884492a90cede8883e5745"
format_stamp: "Formatted at 2026-09-27 18:32:30 on dist-test-slave-n3p4"
server_key: "a5915d35416221ce1e34a536f1be079e"
server_key_iv: "e2acaca0ac8ec74ab2cf19afad451025"
server_key_version: "encryptionkey@0"
I20260927 18:32:30.911820 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "1ec1a10b8c884492a90cede8883e5745"
format_stamp: "Formatted at 2026-09-27 18:32:30 on dist-test-slave-n3p4"
server_key: "a5915d35416221ce1e34a536f1be079e"
server_key_iv: "e2acaca0ac8ec74ab2cf19afad451025"
server_key_version: "encryptionkey@0"
I20260927 18:32:30.916841 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:30.920398 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:30.921829 3436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:30.922034 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "1ec1a10b8c884492a90cede8883e5745"
format_stamp: "Formatted at 2026-09-27 18:32:30 on dist-test-slave-n3p4"
server_key: "a5915d35416221ce1e34a536f1be079e"
server_key_iv: "e2acaca0ac8ec74ab2cf19afad451025"
server_key_version: "encryptionkey@0"
I20260927 18:32:30.922235 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:30.941016 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:30.944665 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:30.944931 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:30.962467 3436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44461
I20260927 18:32:30.962491 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44461 every 8 connection(s)
I20260927 18:32:30.963802 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:30.967403 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3436
I20260927 18:32:30.967651 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:30.967653 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:30.968402 389 external_mini_cluster.cc:1428] Setting key 8fbb771f6b480be4341e8f1cdb942db4
I20260927 18:32:30.980360 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745: Bootstrap starting.
Sep 27 18:32:30 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:30.983569 3503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:30.984560 3503 log.cc:824] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:30.987502 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745: No bootstrap required, opened a new log
I20260927 18:32:30.991384 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:30.970491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.97.126:44461)
0927 18:32:30.971373 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:30.971384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:30.971423 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:32:30.973882 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:30.973905 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:30.974352 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:30.974718 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:30.976346 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:30.977724 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:30.979125 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:30.979806 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:30.980027 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:30.984125 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:30.984133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:30.984147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:30.984184 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:30.987457 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:30.988167 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:30.988185 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:30.988191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:30.988348 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:30.988739 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:30.988744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:30.988749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:30.989258 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:30.989590 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:30.990049 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 18:32:30.990434 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":450,"threads_started":1}
I20260927 18:32:30.995293 3503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } }
I20260927 18:32:30.995627 3503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:30.995730 3503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec1a10b8c884492a90cede8883e5745, State: Initialized, Role: FOLLOWER
I20260927 18:32:30.996280 3503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } }
I20260927 18:32:30.996510 3503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:30.996639 3503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:30.996814 3503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:30.999285 3503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } }
I20260927 18:32:30.999766 3503 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec1a10b8c884492a90cede8883e5745; no voters:
I20260927 18:32:31.000303 3503 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:31.000437 3508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:31.001865 3508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec1a10b8c884492a90cede8883e5745, State: Running, Role: LEADER
I20260927 18:32:31.002461 3503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:31.002365 3508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } }
I20260927 18:32:31.005270 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec1a10b8c884492a90cede8883e5745. Latest consensus state: current_term: 1 leader_uuid: "1ec1a10b8c884492a90cede8883e5745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } } }
I20260927 18:32:31.005472 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec1a10b8c884492a90cede8883e5745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec1a10b8c884492a90cede8883e5745" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44461 } } }
I20260927 18:32:31.005867 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:31.005697 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:31.011461 3517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:31.016623 3517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:31.025527 3517 catalog_manager.cc:1347] Generated new cluster ID: 6fd2bef359684770b5e8a0d576f742e4
I20260927 18:32:31.025647 3517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:31.075642 3517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:31.077518 3517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:31.089919 3517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec1a10b8c884492a90cede8883e5745: Generated new TSK 0
I20260927 18:32:31.090993 3517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:31.145689 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:31.482348 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:31.482652 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:31.482826 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:31.490381 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:31.490551 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:31.490734 3529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:31.501024 3529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:31.502489 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:31.503886 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:31.517134 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:31.517170 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:31.517216 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:31.518044 3529 server_base.cc:1034] running on GCE node
I20260927 18:32:31.518713 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:31.519688 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:31.520919 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790533951520857 us; error 61 us; skew 500 ppm
I20260927 18:32:31.521270 3529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:31 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:31.524530 3529 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:31.526177 3529 webserver.cc:460] Webserver started at http://127.0.97.65:43499/ using document root <none> and password file <none>
I20260927 18:32:31.526958 3529 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:31.527100 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:31.527401 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:31.530812 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "56a90b0366ef4d19aee319b0f8a4b589"
format_stamp: "Formatted at 2026-09-27 18:32:31 on dist-test-slave-n3p4"
server_key: "93f6a61bc07c674fb6e7f43025cd9f95"
server_key_iv: "a5469e9697bebeee75d23addda19f710"
server_key_version: "encryptionkey@0"
I20260927 18:32:31.531627 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "56a90b0366ef4d19aee319b0f8a4b589"
format_stamp: "Formatted at 2026-09-27 18:32:31 on dist-test-slave-n3p4"
server_key: "93f6a61bc07c674fb6e7f43025cd9f95"
server_key_iv: "a5469e9697bebeee75d23addda19f710"
server_key_version: "encryptionkey@0"
I20260927 18:32:31.536741 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:31.540242 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:31.541620 3529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:31.541852 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "56a90b0366ef4d19aee319b0f8a4b589"
format_stamp: "Formatted at 2026-09-27 18:32:31 on dist-test-slave-n3p4"
server_key: "93f6a61bc07c674fb6e7f43025cd9f95"
server_key_iv: "a5469e9697bebeee75d23addda19f710"
server_key_version: "encryptionkey@0"
I20260927 18:32:31.542054 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:31.565034 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:31.569351 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:31.569669 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:31.570672 3529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:31.571991 3529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:31.572098 3529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:31.572243 3529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:31.572314 3529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:31.606135 3529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34763
I20260927 18:32:31.606196 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34763 every 8 connection(s)
I20260927 18:32:31.607636 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:31.614283 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3529
I20260927 18:32:31.614501 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:31.615104 389 external_mini_cluster.cc:1428] Setting key b9dc8c31ea564d659ccdde1a0fe7b5bf
Sep 27 18:32:31 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:31.628654 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:31.610395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36955 (local address 127.0.97.126:44461)
0927 18:32:31.610825 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:31.610830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:31.611950 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 18:32:31.613734 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:31.613738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:31.613862 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:31.613995 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:31.615600 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:31.616363 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:31.618226 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:31.619052 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:31.619198 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:31.622854 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:31.622859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:31.622863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:31.622891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:31.625468 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:31.626382 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:31.626389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:31.626392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:31.626516 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:31.626955 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:31.626960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:31.626965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:31.627224 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:31.627372 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:31.628071 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 18:32:31.628283 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260927 18:32:31.629611 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:31.610972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44461 (local address 127.0.97.65:36955)
0927 18:32:31.611786 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:31.611837 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:31.613489 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:31.614181 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:31.614189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:31.614793 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:31.615395 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:31.615412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:31.616515 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:31.616520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:31.618080 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:31.618091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:31.619261 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:31.619266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:31.619452 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:31.620199 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:31.620225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:31.622638 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:31.625618 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:31.625628 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:31.625644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:31.626263 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:31.626639 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:31.626645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:31.626649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:31.626836 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:31.627347 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:31.627356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:31.627701 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 18:32:31.627962 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 18:32:31.628495 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":338,"threads_started":1}
I20260927 18:32:31.631143 3657 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44461
I20260927 18:32:31.631443 3657 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:31.632143 3657 heartbeater.cc:511] Master 127.0.97.126:44461 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:32:31.634445 3468 ts_manager.cc:194] Registered new tserver with Master: 56a90b0366ef4d19aee319b0f8a4b589 (127.0.97.65:34763)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:32:31.636042 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:36955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:31.658233 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:31.978322 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:31.978564 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:31.978729 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:31.985934 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:31.986032 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:31.986222 3663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:31.995874 3663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:31.997148 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:31.998479 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:32.011672 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:32.011735 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:32.011699 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:32.012835 3663 server_base.cc:1034] running on GCE node
I20260927 18:32:32.013497 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:32.014614 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:32.015834 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790533952015785 us; error 47 us; skew 500 ppm
I20260927 18:32:32.016171 3663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:32.019414 3663 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:32.020885 3663 webserver.cc:460] Webserver started at http://127.0.97.66:40035/ using document root <none> and password file <none>
I20260927 18:32:32.021582 3663 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:32.021723 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:32.022046 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:32.024914 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "42e44e1bf05240a9937381cf810ba6c8"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "a57fb014310f6ad0b3e81a1df54aaf98"
server_key_iv: "5a542c4c4165e01a068dd4725afde623"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.025722 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "42e44e1bf05240a9937381cf810ba6c8"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "a57fb014310f6ad0b3e81a1df54aaf98"
server_key_iv: "5a542c4c4165e01a068dd4725afde623"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.030800 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:32:32.034001 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.035364 3663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:32:32.035521 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "42e44e1bf05240a9937381cf810ba6c8"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "a57fb014310f6ad0b3e81a1df54aaf98"
server_key_iv: "5a542c4c4165e01a068dd4725afde623"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.035717 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:32.051883 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:32.055965 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:32.056253 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:32.057189 3663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:32.058542 3663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:32.058620 3663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.058813 3663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:32.058872 3663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.091524 3663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:40093
I20260927 18:32:32.091609 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:40093 every 8 connection(s)
I20260927 18:32:32.092790 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:32.096019 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3663
I20260927 18:32:32.096215 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:32.096781 389 external_mini_cluster.cc:1428] Setting key 8f559a3e1b2540fa99c23037df6085b2
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:32.113963 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.095406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57909 (local address 127.0.97.126:44461)
0927 18:32:32.095604 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.095609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.096925 (+ 1316us) server_negotiation.cc:407] Connection header received
0927 18:32:32.098969 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.098975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.099051 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.099167 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.100541 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.101659 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.103329 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.103966 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.104059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.108282 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.108289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.108294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.108325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.111059 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.111746 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.111751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.111756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.111881 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.112298 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.112303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.112307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.112567 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.112712 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.113381 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.113593 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:32.115006 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.096005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44461 (local address 127.0.97.66:57909)
0927 18:32:32.096759 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:32.096802 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:32.098724 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:32.099352 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:32.099360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:32.099847 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:32.100377 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:32.100393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.101879 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.101882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:32.103184 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:32.103194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.104190 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.104199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:32.104510 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.105304 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:32.105337 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:32.108051 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:32.111187 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:32.111196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:32.111213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:32.111621 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:32.111986 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:32.111992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:32.111995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:32.112175 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:32.112686 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:32.112694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:32.113039 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 18:32:32.113298 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 18:32:32.113853 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":300,"threads_started":1}
I20260927 18:32:32.116679 3791 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44461
I20260927 18:32:32.117029 3791 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260927 18:32:32.117908 3791 heartbeater.cc:511] Master 127.0.97.126:44461 requested a full tablet report, sending...
I20260927 18:32:32.119462 3468 ts_manager.cc:194] Registered new tserver with Master: 42e44e1bf05240a9937381cf810ba6c8 (127.0.97.66:40093)
I20260927 18:32:32.120862 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:57909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:32.141686 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:32.464257 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:32.464535 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:32.464628 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:32.471920 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:32.472023 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:32.472133 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:32.481525 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:32.482811 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:32.484071 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:32.498219 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:32.498241 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:32.498935 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:32.499174 3796 server_base.cc:1034] running on GCE node
I20260927 18:32:32.499809 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:32.500810 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:32.502049 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790533952501989 us; error 80 us; skew 500 ppm
I20260927 18:32:32.502398 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:32.505661 3796 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:32.507154 3796 webserver.cc:460] Webserver started at http://127.0.97.67:40515/ using document root <none> and password file <none>
I20260927 18:32:32.507835 3796 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:32.507977 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:32.508298 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:32.511245 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "e297736316f64336886e29ace16a12b3"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "102d47433d107d0822c394ed9e3d7a4c"
server_key_iv: "25d5e8c1bae67e261f8480f3d5f657e6"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.511948 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "e297736316f64336886e29ace16a12b3"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "102d47433d107d0822c394ed9e3d7a4c"
server_key_iv: "25d5e8c1bae67e261f8480f3d5f657e6"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.516935 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:32:32.520439 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.521966 3796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:32.522125 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "e297736316f64336886e29ace16a12b3"
format_stamp: "Formatted at 2026-09-27 18:32:32 on dist-test-slave-n3p4"
server_key: "102d47433d107d0822c394ed9e3d7a4c"
server_key_iv: "25d5e8c1bae67e261f8480f3d5f657e6"
server_key_version: "encryptionkey@0"
I20260927 18:32:32.522320 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:32.540115 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:32.544487 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:32.544786 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:32.545862 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:32.547294 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:32.547379 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.547503 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:32.547559 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:32.580009 3796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:32873
I20260927 18:32:32.580050 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:32873 every 8 connection(s)
I20260927 18:32:32.581247 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:32.590167 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3796
I20260927 18:32:32.590382 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:32.590895 389 external_mini_cluster.cc:1428] Setting key 3a076d69173a572208e9bec7b4175066
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:32.601183 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.583978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:32835 (local address 127.0.97.126:44461)
0927 18:32:32.584170 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.584175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.585256 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 18:32:32.587147 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.587153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.587239 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.587349 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.588797 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.589542 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.591290 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.591874 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.592004 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.595995 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.596000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.596003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.596033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.598629 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.599300 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.599303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.599305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.599418 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.599824 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.599827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.599829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.600067 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.600175 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.600787 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.600934 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:32:32.602208 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.584403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44461 (local address 127.0.97.67:32835)
0927 18:32:32.585101 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:32.585145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:32.586864 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:32.587577 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:32.587585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:32.588088 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:32.588618 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:32.588648 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.589696 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.589709 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:32.591149 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:32.591161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.592133 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.592143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:32.592497 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.593370 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:32.593406 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:32.595816 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:32.598751 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:32.598757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:32.598770 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:32.599186 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:32.599539 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:32.599544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:32.599548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:32.599723 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:32.600166 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:32.600171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:32.600476 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 18:32:32.600693 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 18:32:32.601056 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":273,"threads_started":1}
I20260927 18:32:32.603662 3924 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44461
I20260927 18:32:32.603932 3924 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:32.604758 3924 heartbeater.cc:511] Master 127.0.97.126:44461 requested a full tablet report, sending...
I20260927 18:32:32.606454 3467 ts_manager.cc:194] Registered new tserver with Master: e297736316f64336886e29ace16a12b3 (127.0.97.67:32873)
I20260927 18:32:32.607297 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:32835
I20260927 18:32:32.618798 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:32.639118 3657 heartbeater.cc:503] Master 127.0.97.126:44461 was elected leader, sending a full tablet report...
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:32.665548 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.651985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.0.97.126:44461)
0927 18:32:32.652245 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.652252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.652595 (+ 343us) server_negotiation.cc:407] Connection header received
0927 18:32:32.652754 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.652758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.652821 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.652907 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.654402 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.655106 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.656416 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.656791 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.656876 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.660825 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.660830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.660833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.660865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.663373 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.663950 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.663953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.663956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.664078 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.664419 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.664422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.664424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.664759 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.664932 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.665151 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.665297 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 18:32:32.666144 3467 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60506
I20260927 18:32:32.667953 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3529
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:32.682673 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.668902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.0.97.65:34763)
0927 18:32:32.669511 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.669526 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.669549 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:32.669603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.669609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.669876 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.670011 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.670965 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.671928 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.673003 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.673558 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.673652 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.677265 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.677275 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.677289 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.677319 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.680225 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.680801 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.680808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.680811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.680934 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.681249 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.681252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.681254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.681587 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.681899 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.682108 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.682328 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260927 18:32:32.684092 3652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39288
W20260927 18:32:32.685382 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:32.685588 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3529
I20260927 18:32:32.703004 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3663
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:32.719604 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.703952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.97.66:40093)
0927 18:32:32.704513 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.704520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.704547 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:32.704619 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.704625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.704889 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.705022 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.706600 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.707647 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.708963 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.709574 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.709677 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.714161 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.714172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.714186 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.714218 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.717095 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.717706 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.717713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.717715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.717880 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.718285 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.718288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.718290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.718604 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.718825 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.719128 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.719320 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
W20260927 18:32:32.720986 3786 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41570
W20260927 18:32:32.722108 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:32.722296 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3663
I20260927 18:32:32.738585 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3796
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533952, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:32.756094 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:32.739507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.0.97.67:32873)
0927 18:32:32.740183 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:32.740190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:32.740221 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:32:32.740285 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:32.740291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:32.740515 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:32.740715 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:32.742276 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.743414 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.744749 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:32.745367 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:32.745510 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:32.750620 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:32.750632 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:32.750647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:32.750679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:32.753464 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.754167 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.754174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.754176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.754321 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:32.754651 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:32.754654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:32.754656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:32.754962 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:32.755173 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:32.755571 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:32:32.755781 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":312,"threads_started":1}
W20260927 18:32:32.757385 3919 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36056
W20260927 18:32:32.758667 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:32:32.758827 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3796
I20260927 18:32:32.774325 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3436
I20260927 18:32:32.775465 3498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:32.880164 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3436
2026-09-27T18:32:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3611 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3941](info): set up 2 sockets
Sep 27 18:32:32 dist-test-slave-n3p4 krb5kdc[3941](info): commencing operation
krb5kdc: starting...
W20260927 18:32:33.982246 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:34 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:34.175634 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42483
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:34.499755 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:34.500123 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:34.500298 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:34.508497 3954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:34.508646 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:34.508761 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:34.508852 3954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:34.508947 3954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:34.519923 3954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:34.521345 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:34.522765 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:34.535946 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:34.535897 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:34.535969 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:34.537189 3954 server_base.cc:1034] running on GCE node
I20260927 18:32:34.538060 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:34.539482 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:34.540725 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790533954540685 us; error 35 us; skew 500 ppm
I20260927 18:32:34.541100 3954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:34 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:34.548204 3954 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:34.549642 3954 webserver.cc:460] Webserver started at http://127.0.97.126:45745/ using document root <none> and password file <none>
I20260927 18:32:34.550413 3954 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:34.550561 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:34.550881 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:34.553762 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "164f3a49dc994a30b6712a2a72426875"
format_stamp: "Formatted at 2026-09-27 18:32:34 on dist-test-slave-n3p4"
server_key: "e4e09dc80679d6ef6ac2eb87a152ad42"
server_key_iv: "a62a7789737fc76154c235c6032d6ba3"
server_key_version: "encryptionkey@0"
I20260927 18:32:34.554658 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "164f3a49dc994a30b6712a2a72426875"
format_stamp: "Formatted at 2026-09-27 18:32:34 on dist-test-slave-n3p4"
server_key: "e4e09dc80679d6ef6ac2eb87a152ad42"
server_key_iv: "a62a7789737fc76154c235c6032d6ba3"
server_key_version: "encryptionkey@0"
I20260927 18:32:34.559655 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:34.562983 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:34.564323 3954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:32:34.564512 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "164f3a49dc994a30b6712a2a72426875"
format_stamp: "Formatted at 2026-09-27 18:32:34 on dist-test-slave-n3p4"
server_key: "e4e09dc80679d6ef6ac2eb87a152ad42"
server_key_iv: "a62a7789737fc76154c235c6032d6ba3"
server_key_version: "encryptionkey@0"
I20260927 18:32:34.564713 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:34.591907 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:34.596419 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:34.596715 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:34.615203 3954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42483
I20260927 18:32:34.615223 4020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42483 every 8 connection(s)
I20260927 18:32:34.616664 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:34.620329 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:34.623129 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 3954
I20260927 18:32:34.623338 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:34.623958 389 external_mini_cluster.cc:1428] Setting key cecab7e22c53fcc540e8c1ad8b788768
I20260927 18:32:34.633293 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875: Bootstrap starting.
I20260927 18:32:34.636807 4021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:34.637852 4021 log.cc:824] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:32:34 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:34.640949 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875: No bootstrap required, opened a new log
I20260927 18:32:34.647621 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:34.626002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.0.97.126:42483)
0927 18:32:34.626808 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:34.626824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:34.626872 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:32:34.628997 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:34.629028 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:34.629596 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:34.630063 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:34.631342 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:34.633232 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:34.634633 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:34.635758 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:34.635950 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:34.640166 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:34.640174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:34.640188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:34.640226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:34.643465 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:34.644283 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:34.644288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:34.644298 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:34.644455 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:34.644879 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:34.644883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:34.644885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:34.645358 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:34.645631 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:34.646124 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 18:32:34.646580 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":315,"threads_started":1}
I20260927 18:32:34.649732 4021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } }
I20260927 18:32:34.650146 4021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:34.650254 4021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164f3a49dc994a30b6712a2a72426875, State: Initialized, Role: FOLLOWER
I20260927 18:32:34.650892 4021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } }
I20260927 18:32:34.651124 4021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:34.651273 4021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:34.651443 4021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:34.653514 4021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } }
I20260927 18:32:34.654035 4021 leader_election.cc:302] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164f3a49dc994a30b6712a2a72426875; no voters:
I20260927 18:32:34.654526 4021 leader_election.cc:288] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:34.654739 4026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:34.656082 4026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [term 1 LEADER]: Becoming Leader. State: Replica: 164f3a49dc994a30b6712a2a72426875, State: Running, Role: LEADER
I20260927 18:32:34.656649 4021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:34.656589 4026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } }
I20260927 18:32:34.659590 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164f3a49dc994a30b6712a2a72426875. Latest consensus state: current_term: 1 leader_uuid: "164f3a49dc994a30b6712a2a72426875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } } }
I20260927 18:32:34.660064 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:34.659565 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164f3a49dc994a30b6712a2a72426875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164f3a49dc994a30b6712a2a72426875" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42483 } } }
I20260927 18:32:34.660326 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:34.662281 4035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:34.667796 4035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:34.676313 4035 catalog_manager.cc:1347] Generated new cluster ID: 3f503081b14741f7b49de0937ae8cf37
I20260927 18:32:34.676420 4035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:34.723246 4035 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:34.724618 4035 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:34.735212 4035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164f3a49dc994a30b6712a2a72426875: Generated new TSK 0
I20260927 18:32:34.736137 4035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:34.785697 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:35.108855 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:35.109162 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:35.109323 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:35.116513 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:35.116617 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:35.116792 4047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:35.126633 4047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:35.127992 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:35.129307 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:35.142803 4055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:35.142942 4052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:35.142876 4053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:35.144057 4047 server_base.cc:1034] running on GCE node
I20260927 18:32:35.144748 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:35.145750 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:35.146998 4047 hybrid_clock.cc:648] HybridClock initialized: now 1790533955146940 us; error 59 us; skew 500 ppm
I20260927 18:32:35.147385 4047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:35 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:35.150911 4047 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:35.152482 4047 webserver.cc:460] Webserver started at http://127.0.97.65:45823/ using document root <none> and password file <none>
I20260927 18:32:35.153198 4047 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:35.153344 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:35.153653 4047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:35.156664 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "9844b87ff66440eda82e45f114a9d500"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "7f31f6d62f1fa2187da5035d8bfc9639"
server_key_iv: "c99a27f04f013c7a2b6b92b0586857a8"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.157469 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "9844b87ff66440eda82e45f114a9d500"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "7f31f6d62f1fa2187da5035d8bfc9639"
server_key_iv: "c99a27f04f013c7a2b6b92b0586857a8"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.162714 4047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:32:35.166095 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.167560 4047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:32:35.167783 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "9844b87ff66440eda82e45f114a9d500"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "7f31f6d62f1fa2187da5035d8bfc9639"
server_key_iv: "c99a27f04f013c7a2b6b92b0586857a8"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.167950 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:35.180240 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:35.184367 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:35.184598 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:35.185456 4047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:35.186765 4047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:35.186836 4047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.186916 4047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:35.186964 4047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.222641 4047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42923
I20260927 18:32:35.222674 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42923 every 8 connection(s)
I20260927 18:32:35.224027 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:35.234357 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4047
I20260927 18:32:35.234570 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:35.235183 389 external_mini_cluster.cc:1428] Setting key 551bdcfc05358832578f2977a1d6bc13
Sep 27 18:32:35 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:35.245599 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:35.226733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:38363 (local address 127.0.97.126:42483)
0927 18:32:35.227189 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:35.227195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:35.228188 (+ 993us) server_negotiation.cc:407] Connection header received
0927 18:32:35.230021 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:35.230026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:35.230108 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:35.230213 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:35.231683 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.232425 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.234114 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.234611 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.234777 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:35.238988 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:35.238993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:35.238997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:35.239029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:35.241704 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:35.242725 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:35.242734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:35.242739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:35.242926 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:35.243371 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:35.243379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:35.243383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:35.243848 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:35.244031 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:35.244703 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 18:32:35.244993 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260927 18:32:35.246122 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:35.227295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42483 (local address 127.0.97.65:38363)
0927 18:32:35.228026 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:35.228063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:35.229655 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:35.230416 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:35.230424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:35.230937 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:35.231517 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:35.231534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.232582 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.232586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:35.233954 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:35.233969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.234968 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.234975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:35.235238 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:35.236008 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:35.236044 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:35.238753 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:35.241884 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:35.241897 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:35.241918 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:35.242559 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:35.243051 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:35.243054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:35.243056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:35.243233 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:35.243994 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:35.244001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:35.244314 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 18:32:35.244545 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 18:32:35.244936 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":294,"threads_started":1}
I20260927 18:32:35.247725 4175 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42483
I20260927 18:32:35.248102 4175 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:35.249140 4175 heartbeater.cc:511] Master 127.0.97.126:42483 requested a full tablet report, sending...
I20260927 18:32:35.251689 3986 ts_manager.cc:194] Registered new tserver with Master: 9844b87ff66440eda82e45f114a9d500 (127.0.97.65:42923)
I20260927 18:32:35.253194 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:38363
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:35.285445 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:35.604498 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:35.604775 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:35.604867 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:35.611969 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:35.612079 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:35.612274 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:35.626645 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:35.628165 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:35.629756 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:35.644146 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:35.644223 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:35.644225 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:35.645347 4181 server_base.cc:1034] running on GCE node
I20260927 18:32:35.646117 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:35.647495 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:35.648710 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790533955648647 us; error 62 us; skew 500 ppm
I20260927 18:32:35.649061 4181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:35 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:35.652388 4181 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:35.653820 4181 webserver.cc:460] Webserver started at http://127.0.97.66:38839/ using document root <none> and password file <none>
I20260927 18:32:35.654503 4181 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:35.654608 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:35.654871 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:35.658048 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "ad527410b1a94436a7f5f3b80dfaa0a4"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "9a46a6ddc05d2b22dec806f0f082e4a2"
server_key_iv: "ad9ec3f1423d827ec7f7e526e637605f"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.658813 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "ad527410b1a94436a7f5f3b80dfaa0a4"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "9a46a6ddc05d2b22dec806f0f082e4a2"
server_key_iv: "ad9ec3f1423d827ec7f7e526e637605f"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.664438 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:35.667991 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.669438 4181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:35.669657 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "ad527410b1a94436a7f5f3b80dfaa0a4"
format_stamp: "Formatted at 2026-09-27 18:32:35 on dist-test-slave-n3p4"
server_key: "9a46a6ddc05d2b22dec806f0f082e4a2"
server_key_iv: "ad9ec3f1423d827ec7f7e526e637605f"
server_key_version: "encryptionkey@0"
I20260927 18:32:35.669926 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:35.685139 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:35.689291 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:35.689589 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:35.690550 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:35.691799 4181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:35.691897 4181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.692046 4181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:35.692116 4181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:35.724913 4181 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33693
I20260927 18:32:35.724951 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33693 every 8 connection(s)
I20260927 18:32:35.726240 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:35.734396 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4181
I20260927 18:32:35.734570 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:35.735102 389 external_mini_cluster.cc:1428] Setting key b06c8cf7ea770108f4e22cdadaa8ce88
Sep 27 18:32:35 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:35.750561 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:35.728953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43015 (local address 127.0.97.126:42483)
0927 18:32:35.729170 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:35.729175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:35.730408 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 18:32:35.732443 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:35.732449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:35.732540 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:35.732665 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:35.734447 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.735355 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.738213 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.738944 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.739064 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:35.743601 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:35.743608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:35.743613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:35.743660 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:35.747319 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:35.748028 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:35.748037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:35.748043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:35.748190 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:35.748611 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:35.748619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:35.748625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:35.748991 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:35.749158 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:35.749853 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 18:32:35.750110 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 18:32:35.751468 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:35.729401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42483 (local address 127.0.97.66:43015)
0927 18:32:35.730237 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:35.730282 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:35.732170 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:35.732898 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:35.732907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:35.733409 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:35.734205 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:35.734232 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.735539 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.735544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:35.738046 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:35.738063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:35.739424 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:35.739434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:35.739800 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:35.740539 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:35.740574 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:35.743335 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:35.747453 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:35.747460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:35.747474 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:35.747892 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:35.748293 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:35.748296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:35.748299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:35.748479 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:35.749111 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:35.749118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:35.749430 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 18:32:35.749694 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 18:32:35.750145 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":397,"threads_started":1}
I20260927 18:32:35.753053 4309 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42483
I20260927 18:32:35.753379 4309 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:35.754161 4309 heartbeater.cc:511] Master 127.0.97.126:42483 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:35.755600 3986 ts_manager.cc:194] Registered new tserver with Master: ad527410b1a94436a7f5f3b80dfaa0a4 (127.0.97.66:33693)
I20260927 18:32:35.756651 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:43015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:35.781767 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:36.095808 4314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:36.096077 4314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:36.096169 4314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:36.103152 4314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:36.103245 4314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:36.103353 4314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:36.112603 4314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:36.113734 4314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:36.114946 4314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:36.127166 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:36.127322 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:36.127287 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:36.128177 4314 server_base.cc:1034] running on GCE node
I20260927 18:32:36.128759 4314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:36.129639 4314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:36.130878 4314 hybrid_clock.cc:648] HybridClock initialized: now 1790533956130827 us; error 36 us; skew 500 ppm
I20260927 18:32:36.131188 4314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:36.134276 4314 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:36.135617 4314 webserver.cc:460] Webserver started at http://127.0.97.67:39655/ using document root <none> and password file <none>
I20260927 18:32:36.136260 4314 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:36.136405 4314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:36.136715 4314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:36.139648 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "8e14c7032e3b4b8293d98c83b9a37ebb"
format_stamp: "Formatted at 2026-09-27 18:32:36 on dist-test-slave-n3p4"
server_key: "75723145daaed1b65128ab1d24cd8b09"
server_key_iv: "5a1c378e6ccec8fd71b9fbf02bcb7d34"
server_key_version: "encryptionkey@0"
I20260927 18:32:36.140404 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "8e14c7032e3b4b8293d98c83b9a37ebb"
format_stamp: "Formatted at 2026-09-27 18:32:36 on dist-test-slave-n3p4"
server_key: "75723145daaed1b65128ab1d24cd8b09"
server_key_iv: "5a1c378e6ccec8fd71b9fbf02bcb7d34"
server_key_version: "encryptionkey@0"
I20260927 18:32:36.145262 4314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 18:32:36.148644 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:36.149938 4314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:32:36.150102 4314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "8e14c7032e3b4b8293d98c83b9a37ebb"
format_stamp: "Formatted at 2026-09-27 18:32:36 on dist-test-slave-n3p4"
server_key: "75723145daaed1b65128ab1d24cd8b09"
server_key_iv: "5a1c378e6ccec8fd71b9fbf02bcb7d34"
server_key_version: "encryptionkey@0"
I20260927 18:32:36.150303 4314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:36.166385 4314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:36.170401 4314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:36.170680 4314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:36.171659 4314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:36.172967 4314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:36.173066 4314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:36.173252 4314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:36.173326 4314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:36.207448 4314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:42931
I20260927 18:32:36.207542 4441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:42931 every 8 connection(s)
I20260927 18:32:36.208735 4314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:36.209424 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4314
I20260927 18:32:36.209635 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:36.210203 389 external_mini_cluster.cc:1428] Setting key 5f581b6ff084fb9c7b0281370ee7a123
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:36.228902 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.211599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:37769 (local address 127.0.97.126:42483)
0927 18:32:36.211747 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.211752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.213419 (+ 1667us) server_negotiation.cc:407] Connection header received
0927 18:32:36.215199 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.215203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.215267 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.215355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.216729 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.217447 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.218950 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.219347 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.219441 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.222749 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.222755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.222760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.222800 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.226146 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.226812 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.226817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.226821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.226941 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.227347 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.227352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.227356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.227607 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.227755 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.228363 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.228559 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:32:36.229753 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.212319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42483 (local address 127.0.97.67:37769)
0927 18:32:36.213269 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:36.213320 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:36.214953 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:36.215553 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:36.215562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:36.216035 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:36.216577 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:36.216593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.217582 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.217585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:36.218828 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:36.218839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.219517 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.219521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:36.219704 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.220373 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:36.220394 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:36.222555 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:36.226283 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:36.226291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:36.226305 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:36.226693 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:36.227044 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:36.227050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:36.227054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:36.227234 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:36.227730 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:36.227738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:36.228032 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 18:32:36.228278 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 18:32:36.228753 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":354,"threads_started":1}
I20260927 18:32:36.231083 4442 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42483
I20260927 18:32:36.231455 4442 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:36.232278 4442 heartbeater.cc:511] Master 127.0.97.126:42483 requested a full tablet report, sending...
I20260927 18:32:36.233769 3986 ts_manager.cc:194] Registered new tserver with Master: 8e14c7032e3b4b8293d98c83b9a37ebb (127.0.97.67:42931)
I20260927 18:32:36.234612 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:37769
I20260927 18:32:36.237876 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:36.257357 4175 heartbeater.cc:503] Master 127.0.97.126:42483 was elected leader, sending a full tablet report...
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:36.287592 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.270247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.97.65:42923)
0927 18:32:36.271016 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.271024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.271051 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:36.271115 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.271122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.271405 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.271607 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.273323 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.274604 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.276194 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.276842 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.277010 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.281479 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.281492 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.281509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.281559 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.284508 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.285398 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.285406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.285408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.285571 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.286060 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.286063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.286065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.286424 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.286711 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.287058 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.287263 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":387,"threads_started":1}
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:36.305742 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.292065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.97.126:42483)
0927 18:32:36.292318 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.292325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.292536 (+ 211us) server_negotiation.cc:407] Connection header received
0927 18:32:36.292748 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.292754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.292829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.292932 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.294230 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.295041 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.296247 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.296684 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.296815 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.300716 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.300721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.300725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.300757 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.303179 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.303783 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.303790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.303797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.303919 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.304298 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.304303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.304307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.304604 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.304952 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.305236 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.305432 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 18:32:36.306380 3986 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42924
I20260927 18:32:36.307359 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4047
I20260927 18:32:36.319614 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.308319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.97.65:42923)
0927 18:32:36.308516 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.308522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.308732 (+ 210us) server_negotiation.cc:407] Connection header received
0927 18:32:36.308924 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.308929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.308986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.309104 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.310279 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.310937 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.312128 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.312562 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.312693 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.314866 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.314871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.314874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.314901 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.317221 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.317829 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.317837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.317840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.317961 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.318323 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.318328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.318331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.318560 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.318803 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.319093 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.319296 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:32:36.320231 4170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:36.406467 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4047
I20260927 18:32:36.423110 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4181
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:36.439723 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.424011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.97.66:33693)
0927 18:32:36.424614 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.424622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.424649 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:36.424718 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.424725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.425127 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.425281 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.426683 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.428058 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.429297 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.429975 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.430085 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.434034 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.434047 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.434066 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.434100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.436989 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.437690 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.437697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.437699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.437891 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.438220 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.438223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.438225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.438554 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.438773 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.439103 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.439298 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
I20260927 18:32:36.440980 4304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:36.531716 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4181
I20260927 18:32:36.547760 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4314
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:36.566020 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:36.548882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.97.67:42931)
0927 18:32:36.549850 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:36.549862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:36.549904 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:32:36.549983 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:36.549992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:36.550249 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:36.550387 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:36.551569 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.552844 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.554631 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:36.555217 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:36.555358 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:36.560002 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:36.560013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:36.560035 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:36.560071 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:36.563092 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.563754 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.563763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.563767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.563892 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:36.564301 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:36.564308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:36.564311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:36.564656 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:36.564957 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:36.565264 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 18:32:36.565537 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":491,"threads_started":1}
I20260927 18:32:36.567386 4437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:36.650578 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4314
I20260927 18:32:36.666679 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 3954
I20260927 18:32:36.668026 4016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:36.771663 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 3954
2026-09-27T18:32:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3891 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[4461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[4461](info): set up 2 sockets
Sep 27 18:32:36 dist-test-slave-n3p4 krb5kdc[4461](info): commencing operation
krb5kdc: starting...
W20260927 18:32:37.874303 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:37 dist-test-slave-n3p4 krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:38.070062 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46373
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:38.399773 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:38.400038 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:38.400203 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:38.407217 4474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:38.407366 4474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:38.407481 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:38.407591 4474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:38.407682 4474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:38.417116 4474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:38.418394 4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:38.419809 4474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:38.432120 4479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:38.432101 4480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:38.432454 4482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:38.432878 4474 server_base.cc:1034] running on GCE node
I20260927 18:32:38.433719 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:38.435155 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:38.436399 4474 hybrid_clock.cc:648] HybridClock initialized: now 1790533958436355 us; error 33 us; skew 500 ppm
I20260927 18:32:38.436743 4474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:38 dist-test-slave-n3p4 krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:38.440021 4474 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:38.441136 4474 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:38.441221 4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:38.441469 4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:38.444470 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "e5a6416cecf84da7a8b803a2ebdc8e52"
format_stamp: "Formatted at 2026-09-27 18:32:38 on dist-test-slave-n3p4"
server_key: "ab2b6e1f78d4120d392e97ed20d596e6"
server_key_iv: "275b1662110b0cb221663d71352907f9"
server_key_version: "encryptionkey@0"
I20260927 18:32:38.445201 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "e5a6416cecf84da7a8b803a2ebdc8e52"
format_stamp: "Formatted at 2026-09-27 18:32:38 on dist-test-slave-n3p4"
server_key: "ab2b6e1f78d4120d392e97ed20d596e6"
server_key_iv: "275b1662110b0cb221663d71352907f9"
server_key_version: "encryptionkey@0"
I20260927 18:32:38.450220 4474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:32:38.453429 4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:38.454746 4474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:38.454909 4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "e5a6416cecf84da7a8b803a2ebdc8e52"
format_stamp: "Formatted at 2026-09-27 18:32:38 on dist-test-slave-n3p4"
server_key: "ab2b6e1f78d4120d392e97ed20d596e6"
server_key_iv: "275b1662110b0cb221663d71352907f9"
server_key_version: "encryptionkey@0"
I20260927 18:32:38.455222 4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:38.473753 4474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:38.478206 4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:38.478515 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:38.498298 4474 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46373
I20260927 18:32:38.498311 4539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46373 every 8 connection(s)
I20260927 18:32:38.499284 4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:38.502944 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:38.508625 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4474
I20260927 18:32:38.508848 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:38.509648 389 external_mini_cluster.cc:1428] Setting key 8101443552fe38271304bdc70affbccc
I20260927 18:32:38.514513 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: Bootstrap starting.
I20260927 18:32:38.518366 4540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:38.519649 4540 log.cc:824] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:38.522975 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: No bootstrap required, opened a new log
Sep 27 18:32:38 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:38.531160 4540 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } }
I20260927 18:32:38.531553 4540 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:38.531711 4540 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a6416cecf84da7a8b803a2ebdc8e52, State: Initialized, Role: FOLLOWER
I20260927 18:32:38.532433 4540 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } }
I20260927 18:32:38.532727 4540 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:38.532889 4540 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:38.533089 4540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:38.534045 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:38.511877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.0.97.126:46373)
0927 18:32:38.512687 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:38.512698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:38.512741 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:32:38.514525 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:38.514549 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:38.515032 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:38.515401 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:38.517271 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:38.518780 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:38.520628 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:38.521450 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:38.521625 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:38.526353 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:38.526362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:38.526377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:38.526416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:38.529577 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:38.530337 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:38.530353 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:38.530361 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:38.530529 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:38.530920 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:38.530926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:38.530929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:38.531491 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:38.531844 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:38.532283 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 18:32:38.532777 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":383,"threads_started":1}
I20260927 18:32:38.535682 4540 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } }
I20260927 18:32:38.536181 4540 leader_election.cc:302] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a6416cecf84da7a8b803a2ebdc8e52; no voters:
I20260927 18:32:38.536851 4540 leader_election.cc:288] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:38.536957 4545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:38.537633 4545 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [term 1 LEADER]: Becoming Leader. State: Replica: e5a6416cecf84da7a8b803a2ebdc8e52, State: Running, Role: LEADER
I20260927 18:32:38.538259 4545 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } }
I20260927 18:32:38.539178 4540 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:38.541112 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } } }
I20260927 18:32:38.541455 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:38.541723 4546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a6416cecf84da7a8b803a2ebdc8e52. Latest consensus state: current_term: 1 leader_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a6416cecf84da7a8b803a2ebdc8e52" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46373 } } }
I20260927 18:32:38.542232 4546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:38.543438 4552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:38.550037 4552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:38.558943 4552 catalog_manager.cc:1347] Generated new cluster ID: 13ed0684e2214f56ac9414f1b7621646
I20260927 18:32:38.559101 4552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:38.600132 4552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:38.601989 4552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:38.610805 4552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: Generated new TSK 0
I20260927 18:32:38.611649 4552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:38.665854 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:39.008019 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:39.008306 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:39.008502 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:39.015849 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:39.015983 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:39.016189 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:39.026094 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:39.027419 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:39.028777 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:39.041638 4571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:39.041674 4574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:39.041706 4572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:39.042516 4566 server_base.cc:1034] running on GCE node
I20260927 18:32:39.043130 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:39.044039 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:39.045313 4566 hybrid_clock.cc:648] HybridClock initialized: now 1790533959045205 us; error 107 us; skew 500 ppm
I20260927 18:32:39.045709 4566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:39 dist-test-slave-n3p4 krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:39.048769 4566 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:39.049911 4566 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:39.050032 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:39.050345 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:39.053153 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "14fa9588860842c2950e725a3b09dc83"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "5c4d4bf627acec343b0522a46821285d"
server_key_iv: "007378c3b56e67ae4f0c87d3c8d23a81"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.054004 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "14fa9588860842c2950e725a3b09dc83"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "5c4d4bf627acec343b0522a46821285d"
server_key_iv: "007378c3b56e67ae4f0c87d3c8d23a81"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.058694 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:32:39.062196 4580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.063458 4566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:32:39.063632 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "14fa9588860842c2950e725a3b09dc83"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "5c4d4bf627acec343b0522a46821285d"
server_key_iv: "007378c3b56e67ae4f0c87d3c8d23a81"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.063848 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:39.085485 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:39.089540 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:39.089855 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:39.090786 4566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:39.092057 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:39.092175 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.092312 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:39.092398 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.124646 4566 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:35741
I20260927 18:32:39.124729 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:35741 every 8 connection(s)
I20260927 18:32:39.125495 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:39.126729 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4566
I20260927 18:32:39.126920 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:39.127728 389 external_mini_cluster.cc:1428] Setting key 766761dc0d86c61e112f088e420b0277
Sep 27 18:32:39 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:32:39.147248 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:39.128250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49979 (local address 127.0.97.126:46373)
0927 18:32:39.128737 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:39.128741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:39.129829 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 18:32:39.131596 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:39.131601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:39.131698 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:39.131820 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:39.133253 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.134022 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.136122 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.136507 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.136648 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:39.140439 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:39.140444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:39.140447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:39.140546 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:39.143547 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:39.144688 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:39.144692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:39.144702 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:39.144842 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:39.145388 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:39.145392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:39.145394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:39.145875 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:39.146021 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:39.146807 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 18:32:39.146985 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260927 18:32:39.148249 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:39.128828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46373 (local address 127.0.97.65:49979)
0927 18:32:39.129630 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:39.129684 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:39.131330 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:39.132050 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:39.132058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:39.132542 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:39.133089 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:39.133107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.134178 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.134186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:39.135969 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:39.135981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.136723 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.136732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:39.136951 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:39.137610 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:39.137632 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:39.140054 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:39.143710 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:39.143721 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:39.143741 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:39.144494 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:39.144958 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:39.144964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:39.144968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:39.145244 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:39.146014 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:39.146027 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:39.146439 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 18:32:39.146694 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 18:32:39.147114 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:32:39.150367 4693 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46373
I20260927 18:32:39.150750 4693 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:39.151503 4693 heartbeater.cc:511] Master 127.0.97.126:46373 requested a full tablet report, sending...
I20260927 18:32:39.154127 4505 ts_manager.cc:194] Registered new tserver with Master: 14fa9588860842c2950e725a3b09dc83 (127.0.97.65:35741)
I20260927 18:32:39.155849 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:39.174846 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:39.492969 4699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:39.493245 4699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:39.493402 4699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:39.502501 4699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:39.502629 4699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:39.502837 4699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:39.516393 4699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:39.517679 4699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:39.518993 4699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:39.531563 4704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:39.531499 4705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:39.531602 4707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:39.532250 4699 server_base.cc:1034] running on GCE node
I20260927 18:32:39.532848 4699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:39.533743 4699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:39.534987 4699 hybrid_clock.cc:648] HybridClock initialized: now 1790533959534938 us; error 46 us; skew 500 ppm
I20260927 18:32:39.535302 4699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:39 dist-test-slave-n3p4 krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:39.538417 4699 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:39.539734 4699 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:39.539862 4699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:39.540172 4699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:39.543074 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "0d96380743884a8e98b088b6f2eceb72"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "a7096d9dc95ee49298ce1146ca50924e"
server_key_iv: "a53b882882ead55de1e2fb35c19947b0"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.543865 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "0d96380743884a8e98b088b6f2eceb72"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "a7096d9dc95ee49298ce1146ca50924e"
server_key_iv: "a53b882882ead55de1e2fb35c19947b0"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.548825 4699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:39.552201 4713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.553438 4699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:39.553633 4699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "0d96380743884a8e98b088b6f2eceb72"
format_stamp: "Formatted at 2026-09-27 18:32:39 on dist-test-slave-n3p4"
server_key: "a7096d9dc95ee49298ce1146ca50924e"
server_key_iv: "a53b882882ead55de1e2fb35c19947b0"
server_key_version: "encryptionkey@0"
I20260927 18:32:39.553853 4699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:39.570354 4699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:39.574738 4699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:39.575011 4699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:39.575956 4699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:39.577257 4699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:39.577378 4699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.577504 4699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:39.577590 4699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:39.611338 4699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33429
I20260927 18:32:39.611346 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33429 every 8 connection(s)
I20260927 18:32:39.612293 4699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:39.612752 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4699
I20260927 18:32:39.612958 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:39.613536 389 external_mini_cluster.cc:1428] Setting key 8d2347b7e374ceb8b2e43b6ce07ab864
Sep 27 18:32:39 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:39.634898 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:39.614872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46789 (local address 127.0.97.126:46373)
0927 18:32:39.615037 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:39.615042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:39.616547 (+ 1505us) server_negotiation.cc:407] Connection header received
0927 18:32:39.618401 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:39.618409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:39.618501 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:39.618617 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:39.620228 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.620957 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.623612 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.624027 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.624146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:39.628268 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:39.628277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:39.628292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:39.628335 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:39.631823 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:39.632549 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:39.632559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:39.632562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:39.632720 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:39.633115 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:39.633120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:39.633123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:39.633483 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:39.633657 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:39.634463 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 18:32:39.634673 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260927 18:32:39.636160 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:39.615546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46373 (local address 127.0.97.66:46789)
0927 18:32:39.616358 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:39.616408 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:39.618147 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:39.618855 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:39.618863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:39.619385 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:39.620045 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:39.620063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.621125 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.621133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:39.623460 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:39.623476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:39.624268 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:39.624276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:39.624606 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:39.625315 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:39.625348 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:39.628021 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:39.631957 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:39.631964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:39.631976 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:39.632401 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:39.632835 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:39.632838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:39.632841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:39.632997 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:39.633655 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:39.633662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:39.634051 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 18:32:39.634315 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 18:32:39.634755 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
I20260927 18:32:39.637624 4826 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46373
I20260927 18:32:39.637975 4826 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:39.638746 4826 heartbeater.cc:511] Master 127.0.97.126:46373 requested a full tablet report, sending...
I20260927 18:32:39.640579 4505 ts_manager.cc:194] Registered new tserver with Master: 0d96380743884a8e98b088b6f2eceb72 (127.0.97.66:33429)
I20260927 18:32:39.641417 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:46789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:39.660933 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:39.974789 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:39.975044 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:39.975206 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:39.982133 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:39.982225 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:39.982421 4831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:39.991788 4831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:39.993011 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:39.994331 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:40.006940 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:40.006995 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:40.006909 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:40.007632 4831 server_base.cc:1034] running on GCE node
I20260927 18:32:40.008302 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:40.009248 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:40.010531 4831 hybrid_clock.cc:648] HybridClock initialized: now 1790533960010471 us; error 45 us; skew 500 ppm
I20260927 18:32:40.010869 4831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:40 dist-test-slave-n3p4 krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:40.014256 4831 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:40.015431 4831 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:40.015583 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:40.015918 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:40.019006 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2"
format_stamp: "Formatted at 2026-09-27 18:32:40 on dist-test-slave-n3p4"
server_key: "3cf3abb6e127ccfb937f8dcb03c231ec"
server_key_iv: "6dc16cff048d1652cf3c9345034be384"
server_key_version: "encryptionkey@0"
I20260927 18:32:40.019862 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2"
format_stamp: "Formatted at 2026-09-27 18:32:40 on dist-test-slave-n3p4"
server_key: "3cf3abb6e127ccfb937f8dcb03c231ec"
server_key_iv: "6dc16cff048d1652cf3c9345034be384"
server_key_version: "encryptionkey@0"
I20260927 18:32:40.024926 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:40.028427 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:40.029855 4831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:40.030032 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2"
format_stamp: "Formatted at 2026-09-27 18:32:40 on dist-test-slave-n3p4"
server_key: "3cf3abb6e127ccfb937f8dcb03c231ec"
server_key_iv: "6dc16cff048d1652cf3c9345034be384"
server_key_version: "encryptionkey@0"
I20260927 18:32:40.030267 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:40.044870 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:40.049113 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:40.049407 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:40.050464 4831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:40.051879 4831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:40.052011 4831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:40.052171 4831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:40.052280 4831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:40.088356 4831 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38937
I20260927 18:32:40.088469 4957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38937 every 8 connection(s)
I20260927 18:32:40.089290 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:40.098915 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 4831
I20260927 18:32:40.099128 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:40.099568 389 external_mini_cluster.cc:1428] Setting key 16d9819ccb0de6d1b955a7e129e81bc6
Sep 27 18:32:40 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:40.109081 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.091928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46989 (local address 127.0.97.126:46373)
0927 18:32:40.092114 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.092118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.093203 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 18:32:40.095075 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.095081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.095169 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.095277 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:40.096744 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.097471 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.099097 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.099583 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.099716 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.103734 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:40.103738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:40.103741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:40.103776 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:40.106424 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.107148 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.107152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.107155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.107282 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.107668 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.107671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.107673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.107906 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:40.108016 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.108691 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.108844 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:32:40.110241 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.092342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46373 (local address 127.0.97.67:46989)
0927 18:32:40.093042 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.093081 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.094798 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.095495 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.095510 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.095990 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:40.096562 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.096579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.097630 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.097634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.098952 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.098964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.099818 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.099825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.100097 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.100858 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:40.100894 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:40.103528 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:40.106562 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:40.106569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:40.106581 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:40.107022 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:40.107379 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:40.107383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:40.107385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:40.107566 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:40.108021 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:40.108027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:40.108334 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.108586 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.108996 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260927 18:32:40.111861 4958 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46373
I20260927 18:32:40.112145 4958 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:40.112811 4958 heartbeater.cc:511] Master 127.0.97.126:46373 requested a full tablet report, sending...
I20260927 18:32:40.114648 4505 ts_manager.cc:194] Registered new tserver with Master: b8feb6fa891a48f9b247e4e6ff1d8fb2 (127.0.97.67:38937)
I20260927 18:32:40.115808 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:40.115833 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:46989
I20260927 18:32:40.134660 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.121301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.0.97.126:46373)
0927 18:32:40.121486 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.121490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.122095 (+ 605us) server_negotiation.cc:407] Connection header received
0927 18:32:40.122217 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.122221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.122280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.122370 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:40.123712 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.124809 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.126144 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.126879 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.127015 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.129290 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:40.129297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:40.129302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:40.129338 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:40.132058 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.132652 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.132656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.132659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.132778 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.133108 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.133111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.133113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.133479 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:40.133748 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.134061 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.134219 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:40.138641 4505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:40.146286 4505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:40.159884 4693 heartbeater.cc:503] Master 127.0.97.126:46373 was elected leader, sending a full tablet report...
I20260927 18:32:40.178462 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.158386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.0.97.66:33429)
0927 18:32:40.172270 (+ 13884us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.172279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.172305 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:32:40.172396 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.172405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.172636 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.172880 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:40.173465 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.174600 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.176366 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.177271 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.177416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.177623 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.177957 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.178115 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13694,"thread_start_us":13508,"threads_started":1}
I20260927 18:32:40.178560 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.158373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33429 (local address 127.0.0.1:44910)
0927 18:32:40.169394 (+ 11021us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.169453 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.169688 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.172858 (+ 3170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.172864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.172895 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:40.173316 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.173326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.174743 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.174751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.176216 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.176228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.177459 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.177465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.177700 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.177740 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.177917 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.178115 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10604,"mutex_wait_us":57,"thread_start_us":13167,"threads_started":1}
I20260927 18:32:40.179237 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.158402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38937 (local address 127.0.0.1:47044)
0927 18:32:40.170211 (+ 11809us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.170231 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.170379 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.171652 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.171663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.171695 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:40.172094 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.172110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.173905 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.173910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.176247 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.176262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.178401 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.178407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.178605 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.178651 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.178762 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.178908 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11695,"thread_start_us":10426,"threads_started":1}
I20260927 18:32:40.179359 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.159037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.97.67:38937)
0927 18:32:40.159728 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.159737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.171082 (+ 11345us) server_negotiation.cc:407] Connection header received
0927 18:32:40.171200 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.171209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.171452 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.171655 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:40.172282 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.173746 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.176436 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.177896 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.178018 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.178262 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.178852 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.179031 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":412,"threads_started":1}
I20260927 18:32:40.182497 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.162008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35741 (local address 127.0.0.1:55908)
0927 18:32:40.171857 (+ 9849us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.171880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.172008 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.172608 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.172612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.172636 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:40.173081 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.173092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.174998 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.175004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.178140 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.178153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.180797 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.180805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.181919 (+ 1114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.181950 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.182037 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.182149 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9689,"mutex_wait_us":29,"thread_start_us":7950,"threads_started":1}
I20260927 18:32:40.182343 4893 tablet_service.cc:1467] Processing CreateTablet for tablet 611ee2e850444fceb72857c27a50abc5 (DEFAULT_TABLE table=test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:40.182551 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 611ee2e850444fceb72857c27a50abc5 (DEFAULT_TABLE table=test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:40.183247 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.162406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.0.97.65:35741)
0927 18:32:40.170987 (+ 8581us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.170998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.172152 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 18:32:40.172255 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.172262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.172444 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.172625 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:40.173310 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.174828 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.178314 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.180331 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.180450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.180670 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.182782 (+ 2112us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.182927 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8310,"thread_start_us":8167,"threads_started":1}
I20260927 18:32:40.183707 4893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611ee2e850444fceb72857c27a50abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:40.183849 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611ee2e850444fceb72857c27a50abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:40.186292 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 611ee2e850444fceb72857c27a50abc5 (DEFAULT_TABLE table=test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:40.187543 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611ee2e850444fceb72857c27a50abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:40.194645 4976 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Bootstrap starting.
I20260927 18:32:40.198040 4976 tablet_bootstrap.cc:654] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:40.199120 4976 log.cc:824] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:40.199615 4977 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Bootstrap starting.
I20260927 18:32:40.202572 4979 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Bootstrap starting.
I20260927 18:32:40.203188 4977 tablet_bootstrap.cc:654] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:40.203362 4976 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: No bootstrap required, opened a new log
I20260927 18:32:40.203639 4976 ts_tablet_manager.cc:1397] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 18:32:40.204409 4977 log.cc:824] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:40.205551 4979 tablet_bootstrap.cc:654] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:40.206560 4979 log.cc:824] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:40.207734 4977 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: No bootstrap required, opened a new log
I20260927 18:32:40.207996 4977 ts_tablet_manager.cc:1397] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 18:32:40.211866 4979 tablet_bootstrap.cc:492] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: No bootstrap required, opened a new log
I20260927 18:32:40.212113 4979 ts_tablet_manager.cc:1397] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 18:32:40.214367 4976 raft_consensus.cc:348] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.214795 4976 raft_consensus.cc:374] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:40.214895 4976 raft_consensus.cc:729] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8feb6fa891a48f9b247e4e6ff1d8fb2, State: Initialized, Role: FOLLOWER
I20260927 18:32:40.215509 4976 consensus_queue.cc:260] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.216579 4958 heartbeater.cc:503] Master 127.0.97.126:46373 was elected leader, sending a full tablet report...
I20260927 18:32:40.217195 4976 ts_tablet_manager.cc:1428] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 18:32:40.219166 4977 raft_consensus.cc:348] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.219550 4977 raft_consensus.cc:374] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:40.219647 4977 raft_consensus.cc:729] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d96380743884a8e98b088b6f2eceb72, State: Initialized, Role: FOLLOWER
I20260927 18:32:40.220218 4977 consensus_queue.cc:260] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.220217 4979 raft_consensus.cc:348] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.220607 4979 raft_consensus.cc:374] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:40.220710 4979 raft_consensus.cc:729] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fa9588860842c2950e725a3b09dc83, State: Initialized, Role: FOLLOWER
I20260927 18:32:40.221231 4979 consensus_queue.cc:260] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.223886 4826 heartbeater.cc:503] Master 127.0.97.126:46373 was elected leader, sending a full tablet report...
I20260927 18:32:40.224607 4977 ts_tablet_manager.cc:1428] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 18:32:40.226416 4979 ts_tablet_manager.cc:1428] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260927 18:32:40.300241 4982 raft_consensus.cc:482] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:40.300547 4982 raft_consensus.cc:504] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.302412 4982 leader_election.cc:288] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d96380743884a8e98b088b6f2eceb72 (127.0.97.66:33429), 14fa9588860842c2950e725a3b09dc83 (127.0.97.65:35741)
I20260927 18:32:40.310330 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.302891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33429 (local address 127.0.97.67:41843)
0927 18:32:40.303393 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.303421 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.303600 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.303946 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.303952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.303976 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:40.304147 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.304154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.305613 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.305618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.307812 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.307827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.309565 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.309569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.309699 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.309760 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.309889 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.310031 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:32:40.310328 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.302798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41843 (local address 127.0.97.66:33429)
0927 18:32:40.303024 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.303032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.303597 (+ 565us) server_negotiation.cc:407] Connection header received
0927 18:32:40.303776 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.303780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.303843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.303951 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:40.304285 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.305439 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.308003 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.309351 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.309565 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.309675 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.309916 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.310045 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:32:40.310624 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.303017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58427 (local address 127.0.97.65:35741)
0927 18:32:40.303194 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.303199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.303839 (+ 640us) server_negotiation.cc:407] Connection header received
0927 18:32:40.304061 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.304067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.304152 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.304254 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:40.304774 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.305873 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.308263 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.309637 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.309800 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.309935 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.310210 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.310344 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:40.310860 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.303075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35741 (local address 127.0.97.67:58427)
0927 18:32:40.303741 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:40.303755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:40.303883 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:40.304290 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:40.304294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:40.304317 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:40.304602 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.304610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.306019 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.306031 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.308102 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:40.308113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.309872 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.309881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:40.310046 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.310074 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:32:40.310158 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:32:40.310272 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":362,"threads_started":1}
I20260927 18:32:40.311455 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611ee2e850444fceb72857c27a50abc5" candidate_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d96380743884a8e98b088b6f2eceb72" is_pre_election: true
I20260927 18:32:40.311955 4781 raft_consensus.cc:2454] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8feb6fa891a48f9b247e4e6ff1d8fb2 in term 0.
I20260927 18:32:40.311843 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611ee2e850444fceb72857c27a50abc5" candidate_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fa9588860842c2950e725a3b09dc83" is_pre_election: true
I20260927 18:32:40.312454 4648 raft_consensus.cc:2454] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8feb6fa891a48f9b247e4e6ff1d8fb2 in term 0.
I20260927 18:32:40.312762 4848 leader_election.cc:302] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d96380743884a8e98b088b6f2eceb72, b8feb6fa891a48f9b247e4e6ff1d8fb2; no voters:
I20260927 18:32:40.313346 4982 raft_consensus.cc:2781] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:40.313519 4982 raft_consensus.cc:482] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:40.313645 4982 raft_consensus.cc:3037] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:40.315864 4982 raft_consensus.cc:504] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.316542 4982 leader_election.cc:288] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 0d96380743884a8e98b088b6f2eceb72 (127.0.97.66:33429), 14fa9588860842c2950e725a3b09dc83 (127.0.97.65:35741)
I20260927 18:32:40.316975 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611ee2e850444fceb72857c27a50abc5" candidate_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d96380743884a8e98b088b6f2eceb72"
I20260927 18:32:40.316994 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611ee2e850444fceb72857c27a50abc5" candidate_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fa9588860842c2950e725a3b09dc83"
I20260927 18:32:40.317162 4781 raft_consensus.cc:3037] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:40.317199 4648 raft_consensus.cc:3037] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:40.319545 4781 raft_consensus.cc:2454] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8feb6fa891a48f9b247e4e6ff1d8fb2 in term 1.
I20260927 18:32:40.319545 4648 raft_consensus.cc:2454] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8feb6fa891a48f9b247e4e6ff1d8fb2 in term 1.
I20260927 18:32:40.319927 4848 leader_election.cc:302] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d96380743884a8e98b088b6f2eceb72, b8feb6fa891a48f9b247e4e6ff1d8fb2; no voters:
I20260927 18:32:40.320226 4982 raft_consensus.cc:2781] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:40.320713 4982 raft_consensus.cc:686] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 1 LEADER]: Becoming Leader. State: Replica: b8feb6fa891a48f9b247e4e6ff1d8fb2, State: Running, Role: LEADER
I20260927 18:32:40.321249 4982 consensus_queue.cc:237] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } }
I20260927 18:32:40.325824 4504 catalog_manager.cc:5495] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8feb6fa891a48f9b247e4e6ff1d8fb2 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8feb6fa891a48f9b247e4e6ff1d8fb2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 } health_report { overall_health: UNKNOWN } } }
W20260927 18:32:40.340955 4959 tablet.cc:2369] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:40.362941 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.357461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.97.67:38937)
0927 18:32:40.357667 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.357671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.357690 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:32:40.357902 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.357906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.357992 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.358135 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:40.358699 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.359439 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.360853 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.361221 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.361313 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.361869 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:40.362425 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:40.362533 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.362685 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.362747 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 18:32:40.364235 4827 tablet.cc:2369] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:40.371544 4648 raft_consensus.cc:1261] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 1 FOLLOWER]: Refusing update from remote peer b8feb6fa891a48f9b247e4e6ff1d8fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:40.371544 4781 raft_consensus.cc:1261] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 1 FOLLOWER]: Refusing update from remote peer b8feb6fa891a48f9b247e4e6ff1d8fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:40.372574 4986 consensus_queue.cc:1035] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d96380743884a8e98b088b6f2eceb72" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:40.372927 4982 consensus_queue.cc:1035] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14fa9588860842c2950e725a3b09dc83" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:32:40.377926 4694 tablet.cc:2369] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:40.384799 4991 mvcc.cc:205] Tried to move back new op lower bound from 7334027101671055360 to 7334027101480210432. Current Snapshot: MvccSnapshot[applied={T|T < 7334027101671055360}]
I20260927 18:32:40.401155 4505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:40.401453 4505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:40.405030 4505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5a6416cecf84da7a8b803a2ebdc8e52: Sending DeleteTablet for 3 replicas of tablet 611ee2e850444fceb72857c27a50abc5
I20260927 18:32:40.406236 4761 tablet_service.cc:1514] Processing DeleteTablet for tablet 611ee2e850444fceb72857c27a50abc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44910
I20260927 18:32:40.406275 4893 tablet_service.cc:1514] Processing DeleteTablet for tablet 611ee2e850444fceb72857c27a50abc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47044
I20260927 18:32:40.406535 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 611ee2e850444fceb72857c27a50abc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55908
I20260927 18:32:40.408057 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4566
I20260927 18:32:40.418596 5000 tablet_replica.cc:330] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: stopping tablet replica
I20260927 18:32:40.418721 4998 tablet_replica.cc:330] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: stopping tablet replica
I20260927 18:32:40.419154 4998 raft_consensus.cc:2229] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:40.419174 5000 raft_consensus.cc:2229] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:40.419705 4998 raft_consensus.cc:2258] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:40.419713 5000 raft_consensus.cc:2258] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:40.422026 5000 ts_tablet_manager.cc:1905] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:40.422026 4998 ts_tablet_manager.cc:1905] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:40.423482 4999 tablet_replica.cc:330] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: stopping tablet replica
I20260927 18:32:40.423888 4999 raft_consensus.cc:2229] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:40.424523 4999 raft_consensus.cc:2258] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:40.429241 4998 ts_tablet_manager.cc:1918] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:40.429420 4998 log.cc:1196] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/611ee2e850444fceb72857c27a50abc5
I20260927 18:32:40.429548 5000 ts_tablet_manager.cc:1918] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:40.429706 5000 log.cc:1196] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/611ee2e850444fceb72857c27a50abc5
I20260927 18:32:40.430047 4998 ts_tablet_manager.cc:1939] T 611ee2e850444fceb72857c27a50abc5 P 0d96380743884a8e98b088b6f2eceb72: Deleting consensus metadata
I20260927 18:32:40.430225 5000 ts_tablet_manager.cc:1939] T 611ee2e850444fceb72857c27a50abc5 P 14fa9588860842c2950e725a3b09dc83: Deleting consensus metadata
Sep 27 18:32:40 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:40.431951 4490 catalog_manager.cc:4841] TS 14fa9588860842c2950e725a3b09dc83 (127.0.97.65:35741): tablet 611ee2e850444fceb72857c27a50abc5 (table test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]) successfully deleted
I20260927 18:32:40.432142 4999 ts_tablet_manager.cc:1905] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:40.433560 4491 catalog_manager.cc:4841] TS 0d96380743884a8e98b088b6f2eceb72 (127.0.97.66:33429): tablet 611ee2e850444fceb72857c27a50abc5 (table test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]) successfully deleted
I20260927 18:32:40.439265 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.409824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.0.97.65:35741)
0927 18:32:40.410047 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.410057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.419806 (+ 9749us) server_negotiation.cc:407] Connection header received
0927 18:32:40.420448 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.420457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.420566 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.420679 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:40.423050 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.424226 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.425848 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.426522 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.426651 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.432681 (+ 6030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:40.432694 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:40.432712 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:40.432756 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:40.436123 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.436860 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.436876 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.436879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.437014 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.437402 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.437412 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.437414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.437842 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:40.438167 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.438655 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.438876 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:40.439791 4999 ts_tablet_manager.cc:1918] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:40.440006 4688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:40.440029 4999 log.cc:1196] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/611ee2e850444fceb72857c27a50abc5
I20260927 18:32:40.440629 4999 ts_tablet_manager.cc:1939] T 611ee2e850444fceb72857c27a50abc5 P b8feb6fa891a48f9b247e4e6ff1d8fb2: Deleting consensus metadata
I20260927 18:32:40.442865 4489 catalog_manager.cc:4841] TS b8feb6fa891a48f9b247e4e6ff1d8fb2 (127.0.97.67:38937): tablet 611ee2e850444fceb72857c27a50abc5 (table test-table [id=291cbb8ced4b477c8d74e7bb8f861a53]) successfully deleted
I20260927 18:32:40.530221 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4566
I20260927 18:32:40.548602 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4699
Sep 27 18:32:40 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:40.565742 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.549416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.97.66:33429)
0927 18:32:40.549602 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.549608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.549629 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:40.549814 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.549820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.549948 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.550080 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:40.551248 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.552414 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.553669 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.554315 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.554442 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.558852 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:40.558870 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:40.558892 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:40.558933 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:40.562914 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.563613 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.563623 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.563627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.563777 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.564120 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.564124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.564128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.564610 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:40.564885 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.565117 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.565332 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:32:40.566442 4821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:40.660969 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4699
I20260927 18:32:40.677706 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4831
Sep 27 18:32:40 dist-test-slave-n3p4 krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:40.696413 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:40.678571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.0.97.67:38937)
0927 18:32:40.678796 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:40.678803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:40.678830 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:40.678996 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:40.679002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:40.679097 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:40.679216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:40.680584 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.681855 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.683074 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:40.683763 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:40.683904 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:40.688991 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:40.689008 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:40.689030 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:40.689087 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:40.693152 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.693922 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.693932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.693936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.694136 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:40.694598 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:40.694604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:40.694607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:40.695119 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:40.695444 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:40.695713 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 18:32:40.695951 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:32:40.697178 4953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:40.819125 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4831
I20260927 18:32:40.838402 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 4474
I20260927 18:32:40.839598 4535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:40.948488 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 4474
2026-09-27T18:32:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4176 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:40Z Disabled control of system clock
I20260927 18:32:41.014353 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43827
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:32:41.335310 5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:41.335618 5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:41.335767 5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:41.343178 5009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:41.343286 5009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:41.343374 5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:41.343489 5009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:41.343550 5009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:41.353029 5009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:41.354907 5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:41.357415 5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:41.371205 5014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:41.371243 5015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:41.371589 5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:41.372407 5009 server_base.cc:1034] running on GCE node
I20260927 18:32:41.373308 5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:41.374897 5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:41.376149 5009 hybrid_clock.cc:648] HybridClock initialized: now 1790533961376102 us; error 53 us; skew 500 ppm
I20260927 18:32:41.376508 5009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:41.378906 5009 webserver.cc:460] Webserver started at http://127.0.97.126:35279/ using document root <none> and password file <none>
I20260927 18:32:41.379643 5009 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:41.379774 5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:41.380110 5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:41.383118 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "a69af3a2203846aba9b30e418cd52181"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "5d20867e51d323ddac48eb328e33a441"
server_key_iv: "486f5c7a296aa82cd236c7cb2d36918b"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.383920 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "a69af3a2203846aba9b30e418cd52181"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "5d20867e51d323ddac48eb328e33a441"
server_key_iv: "486f5c7a296aa82cd236c7cb2d36918b"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.389140 5009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:32:41.392630 5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:41.393973 5009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:32:41.394145 5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "a69af3a2203846aba9b30e418cd52181"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "5d20867e51d323ddac48eb328e33a441"
server_key_iv: "486f5c7a296aa82cd236c7cb2d36918b"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.394359 5009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:41.399991 5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:41.400727 5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:41.401013 5009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:41.419967 5009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43827
I20260927 18:32:41.420007 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43827 every 8 connection(s)
I20260927 18:32:41.421330 5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:41.422742 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5009
I20260927 18:32:41.422972 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:41.423789 389 external_mini_cluster.cc:1428] Setting key 770aac547bf909f78662c118a4198e6b
I20260927 18:32:41.426645 5075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:41.432624 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:41.426116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.97.126:43827)
0927 18:32:41.427235 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:41.427246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:41.427286 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:32:41.428875 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:41.428896 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:41.429308 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:41.429717 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:41.430845 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:41.430852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:41.430867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:41.430880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:41.430951 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:41.430970 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:41.431230 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:41.431394 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 18:32:41.431691 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":400,"threads_started":1}
I20260927 18:32:41.439149 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: Bootstrap starting.
I20260927 18:32:41.442353 5075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:41.443391 5075 log.cc:824] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:41.446653 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: No bootstrap required, opened a new log
I20260927 18:32:41.454571 5075 raft_consensus.cc:348] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } }
I20260927 18:32:41.455024 5075 raft_consensus.cc:374] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:41.455152 5075 raft_consensus.cc:729] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69af3a2203846aba9b30e418cd52181, State: Initialized, Role: FOLLOWER
I20260927 18:32:41.455739 5075 consensus_queue.cc:260] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } }
I20260927 18:32:41.455969 5075 raft_consensus.cc:388] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:41.456120 5075 raft_consensus.cc:482] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:41.456310 5075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:41.458760 5075 raft_consensus.cc:504] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } }
I20260927 18:32:41.459285 5075 leader_election.cc:302] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a69af3a2203846aba9b30e418cd52181; no voters:
I20260927 18:32:41.459890 5075 leader_election.cc:288] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:41.460052 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:41.460783 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [term 1 LEADER]: Becoming Leader. State: Replica: a69af3a2203846aba9b30e418cd52181, State: Running, Role: LEADER
I20260927 18:32:41.461345 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } }
I20260927 18:32:41.462199 5075 sys_catalog.cc:564] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:41.464653 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a69af3a2203846aba9b30e418cd52181. Latest consensus state: current_term: 1 leader_uuid: "a69af3a2203846aba9b30e418cd52181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } } }
I20260927 18:32:41.465131 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:41.464705 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a69af3a2203846aba9b30e418cd52181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69af3a2203846aba9b30e418cd52181" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43827 } } }
I20260927 18:32:41.465375 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:41.467013 5089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:41.471798 5089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:41.480476 5089 catalog_manager.cc:1347] Generated new cluster ID: acf428f1d3f64973a28eb69f033d5933
I20260927 18:32:41.480579 5089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:41.512167 5089 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:41.512704 5089 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:41.538910 5089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: Generated new TSK 0
I20260927 18:32:41.540086 5089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:32:41.560985 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--builtin_ntp_servers=127.0.97.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:32:41.882934 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:41.883263 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:41.883437 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:41.890760 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:41.890888 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:41.891067 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:41.900285 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:41.901576 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:41.903416 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:41.916635 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:41.916668 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:41.916715 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:41.921985 5099 server_base.cc:1034] running on GCE node
I20260927 18:32:41.922644 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:41.923702 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:41.924899 5099 hybrid_clock.cc:648] HybridClock initialized: now 1790533961924831 us; error 59 us; skew 500 ppm
I20260927 18:32:41.925230 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:41.927510 5099 webserver.cc:460] Webserver started at http://127.0.97.65:39715/ using document root <none> and password file <none>
I20260927 18:32:41.928215 5099 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:41.928325 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:41.928651 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:41.931533 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "96ae805a4bf4a937433de613c2c7ddfd"
server_key_iv: "0ca48ecc8252eb5173f3d40452d35293"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.932334 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "96ae805a4bf4a937433de613c2c7ddfd"
server_key_iv: "0ca48ecc8252eb5173f3d40452d35293"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.937327 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:32:41.940815 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:41.942118 5099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:32:41.942294 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285"
format_stamp: "Formatted at 2026-09-27 18:32:41 on dist-test-slave-n3p4"
server_key: "96ae805a4bf4a937433de613c2c7ddfd"
server_key_iv: "0ca48ecc8252eb5173f3d40452d35293"
server_key_version: "encryptionkey@0"
I20260927 18:32:41.942521 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:41.947877 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:41.948633 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:41.948895 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:41.949859 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:41.951217 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:41.951333 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:41.951464 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:41.951545 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:41.983618 5099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44657
I20260927 18:32:41.983669 5225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44657 every 8 connection(s)
I20260927 18:32:41.984892 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:41.991796 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5099
I20260927 18:32:41.992002 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:41.992599 389 external_mini_cluster.cc:1428] Setting key bc84aa7061de831d6917cc39e8edf7d7
I20260927 18:32:41.993029 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:41.987405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58549 (local address 127.0.97.126:43827)
0927 18:32:41.987805 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:41.987809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:41.989068 (+ 1259us) server_negotiation.cc:407] Connection header received
0927 18:32:41.990746 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:41.990751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:41.990824 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:41.990941 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:41.992290 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:41.992295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:41.992300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:41.992317 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:41.992365 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:41.992376 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:41.992502 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:41.992691 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:32:41.992757 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":194,"threads_started":1}
I20260927 18:32:41.994055 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:41.988023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43827 (local address 127.0.97.65:58549)
0927 18:32:41.988929 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:41.988963 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:41.990495 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:41.991160 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:41.991168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:41.991253 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:41.991976 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:41.991978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:41.992012 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:41.992130 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:41.992467 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:41.992476 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:41.992486 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:32:41.992764 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 18:32:41.993010 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":252,"threads_started":1}
I20260927 18:32:41.995409 5226 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43827
I20260927 18:32:41.995553 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--builtin_ntp_servers=127.0.97.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:32:41.995738 5226 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:41.996301 5226 heartbeater.cc:511] Master 127.0.97.126:43827 requested a full tablet report, sending...
I20260927 18:32:41.998322 5040 ts_manager.cc:194] Registered new tserver with Master: c7fd6b3289c54f309a8d4a6ebe2a7285 (127.0.97.65:44657)
W20260927 18:32:42.310446 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:42.310731 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:42.310877 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:42.318423 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:42.318544 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:42.318722 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:42.327744 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:42.329012 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:42.330719 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:42.343098 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:42.343017 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:42.343066 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:42.343906 5230 server_base.cc:1034] running on GCE node
I20260927 18:32:42.344444 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:42.345381 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:42.346597 5230 hybrid_clock.cc:648] HybridClock initialized: now 1790533962346547 us; error 53 us; skew 500 ppm
I20260927 18:32:42.346933 5230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:42.349275 5230 webserver.cc:460] Webserver started at http://127.0.97.66:38475/ using document root <none> and password file <none>
I20260927 18:32:42.350032 5230 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:42.350157 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:42.350459 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:42.353261 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "c1b4644d513247a7bdcdc601d7d1211b"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "b8942a421d399de86d360a338e33103f"
server_key_iv: "301c9af8b5086a118734d1fa283a2519"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.354090 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "c1b4644d513247a7bdcdc601d7d1211b"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "b8942a421d399de86d360a338e33103f"
server_key_iv: "301c9af8b5086a118734d1fa283a2519"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.358922 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:32:42.362800 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.364205 5230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:32:42.364373 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "c1b4644d513247a7bdcdc601d7d1211b"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "b8942a421d399de86d360a338e33103f"
server_key_iv: "301c9af8b5086a118734d1fa283a2519"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.364602 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:42.370168 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:42.370914 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:42.371178 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:42.372031 5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:42.373365 5230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:42.373497 5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.373615 5230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:42.373699 5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.406296 5230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43785
I20260927 18:32:42.406342 5356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43785 every 8 connection(s)
I20260927 18:32:42.407655 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:42.413686 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5230
I20260927 18:32:42.413892 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:42.414448 389 external_mini_cluster.cc:1428] Setting key 92be00683713b7c2471c2019a4193a15
I20260927 18:32:42.415627 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.410102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44157 (local address 127.0.97.126:43827)
0927 18:32:42.410286 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.410291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.411662 (+ 1371us) server_negotiation.cc:407] Connection header received
0927 18:32:42.413204 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.413208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.413271 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.413402 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.414828 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.414834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.414838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.414856 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.414894 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.414909 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.415033 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.415235 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.415311 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:32:42.416698 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.410620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43827 (local address 127.0.97.66:44157)
0927 18:32:42.411512 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.411553 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.412970 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.413612 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.413620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.413709 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.414468 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.414510 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.414563 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.414656 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.415006 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.415013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.415018 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.415311 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.415561 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":258,"threads_started":1}
I20260927 18:32:42.417608 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--builtin_ntp_servers=127.0.97.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:32:42.418263 5357 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43827
I20260927 18:32:42.418561 5357 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:42.419165 5357 heartbeater.cc:511] Master 127.0.97.126:43827 requested a full tablet report, sending...
I20260927 18:32:42.420713 5040 ts_manager.cc:194] Registered new tserver with Master: c1b4644d513247a7bdcdc601d7d1211b (127.0.97.66:43785)
W20260927 18:32:42.734076 5360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:42.734395 5360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:42.734560 5360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:42.741954 5360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:42.742085 5360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:42.742259 5360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:42.751703 5360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:42.752960 5360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:42.754771 5360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:42.769085 5366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:42.769090 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:42.769131 5368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:42.770632 5360 server_base.cc:1034] running on GCE node
I20260927 18:32:42.771186 5360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:42.772048 5360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:42.773275 5360 hybrid_clock.cc:648] HybridClock initialized: now 1790533962773242 us; error 43 us; skew 500 ppm
I20260927 18:32:42.773599 5360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:42.775918 5360 webserver.cc:460] Webserver started at http://127.0.97.67:44075/ using document root <none> and password file <none>
I20260927 18:32:42.776618 5360 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:42.776744 5360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:42.777055 5360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:42.779884 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "9ff45f2e844d454c8a804ece2ecb7021"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "7563c31a2126afd52560783cbddc56ad"
server_key_iv: "1798ddaa7a615334800ad52cbea63593"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.780690 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "9ff45f2e844d454c8a804ece2ecb7021"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "7563c31a2126afd52560783cbddc56ad"
server_key_iv: "1798ddaa7a615334800ad52cbea63593"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.785547 5360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 18:32:42.788802 5374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.790267 5360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:42.790439 5360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "9ff45f2e844d454c8a804ece2ecb7021"
format_stamp: "Formatted at 2026-09-27 18:32:42 on dist-test-slave-n3p4"
server_key: "7563c31a2126afd52560783cbddc56ad"
server_key_iv: "1798ddaa7a615334800ad52cbea63593"
server_key_version: "encryptionkey@0"
I20260927 18:32:42.790745 5360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:42.796185 5360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:42.796926 5360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:42.797188 5360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:42.798230 5360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:42.799659 5360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:42.799782 5360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.799906 5360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:42.800004 5360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:42.832271 5360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:34539
I20260927 18:32:42.832356 5486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:34539 every 8 connection(s)
I20260927 18:32:42.834197 5360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:42.836113 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5360
I20260927 18:32:42.836284 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:42.836870 389 external_mini_cluster.cc:1428] Setting key 5f49e9300b0c85ff0f4a521697f67c87
I20260927 18:32:42.844379 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.838077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38509 (local address 127.0.97.126:43827)
0927 18:32:42.838274 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.838280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.840613 (+ 2333us) server_negotiation.cc:407] Connection header received
0927 18:32:42.842325 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.842330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.842391 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.842477 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.843667 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.843673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.843677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.843695 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.843732 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.843752 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.843865 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.844053 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.844116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:32:42.845233 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.838060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43827 (local address 127.0.97.67:38509)
0927 18:32:42.840452 (+ 2392us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.840522 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.842065 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.842706 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.842714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.842799 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.843394 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.843397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.843432 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.843513 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.843841 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.843851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.843858 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.844141 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.844378 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":363,"threads_started":1}
I20260927 18:32:42.846491 5487 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43827
I20260927 18:32:42.846726 5487 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:42.847337 5487 heartbeater.cc:511] Master 127.0.97.126:43827 requested a full tablet report, sending...
I20260927 18:32:42.848614 5040 ts_manager.cc:194] Registered new tserver with Master: 9ff45f2e844d454c8a804ece2ecb7021 (127.0.97.67:34539)
I20260927 18:32:42.852362 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:42.858969 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.856436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.97.126:43827)
0927 18:32:42.856559 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.856562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.857090 (+ 528us) server_negotiation.cc:407] Connection header received
0927 18:32:42.857287 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.857290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.857334 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.857427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.858302 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.858307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.858312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.858328 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.858363 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.858374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.858491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.858657 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.858712 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:32:42.862855 5040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:42.870473 5040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:42.888872 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.883157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43785 (local address 127.0.0.1:53614)
0927 18:32:42.884608 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.884626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.884807 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.886725 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.886733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.886765 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.887474 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.887478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.887493 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.887530 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.888083 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.888094 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.888104 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.888284 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.888372 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"thread_start_us":1146,"threads_started":1}
I20260927 18:32:42.888963 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.882625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36686)
0927 18:32:42.884349 (+ 1724us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.884396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.884549 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.885389 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.885402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.885534 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.886581 (+ 1047us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.886588 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.886626 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.886733 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.888147 (+ 1414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.888153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.888159 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.888333 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.888387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":334,"threads_started":1}
I20260927 18:32:42.889348 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.882854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.97.65:44657)
0927 18:32:42.883642 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.883652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.884700 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 18:32:42.884804 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.884817 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.885115 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.886183 (+ 1068us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.887893 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.887902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.887915 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.887941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.887993 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.888014 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.888765 (+ 751us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.888851 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.888956 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":435,"threads_started":1}
I20260927 18:32:42.889734 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.883545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.0.97.66:43785)
0927 18:32:42.884118 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.884129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.884717 (+ 588us) server_negotiation.cc:407] Connection header received
0927 18:32:42.884956 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.884968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.885210 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.885369 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.887736 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.887746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.887762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.887787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.887848 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.887869 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.888030 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.889227 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.889342 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260927 18:32:42.891984 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.884565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34539 (local address 127.0.0.1:46756)
0927 18:32:42.887946 (+ 3381us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.887962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.888043 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.889474 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.889478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.889503 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.890802 (+ 1299us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.890806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.890821 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.890863 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.891378 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.891382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.891389 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.891529 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.891576 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3251,"thread_start_us":3115,"threads_started":1}
I20260927 18:32:42.892892 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.884903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.0.97.67:34539)
0927 18:32:42.888387 (+ 3484us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.888395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.888427 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:32:42.888491 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.888501 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.888704 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.888845 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.891039 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.891049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.891061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.891095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.891150 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.891172 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.891324 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.892431 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.892518 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3328,"thread_start_us":2485,"threads_started":1}
I20260927 18:32:42.893261 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc03f18108e44a698f6717b258559b6 (DEFAULT_TABLE table=test-table [id=4d3f162a943d4a98a7323095cfbfb8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:42.893791 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc03f18108e44a698f6717b258559b6 (DEFAULT_TABLE table=test-table [id=4d3f162a943d4a98a7323095cfbfb8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:42.894786 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc03f18108e44a698f6717b258559b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:42.895148 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc03f18108e44a698f6717b258559b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:42.896440 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc03f18108e44a698f6717b258559b6 (DEFAULT_TABLE table=test-table [id=4d3f162a943d4a98a7323095cfbfb8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:42.898243 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc03f18108e44a698f6717b258559b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:42.905890 5505 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Bootstrap starting.
I20260927 18:32:42.909403 5505 tablet_bootstrap.cc:654] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:42.909929 5506 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Bootstrap starting.
I20260927 18:32:42.910810 5505 log.cc:824] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:42.913275 5506 tablet_bootstrap.cc:654] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:42.914477 5506 log.cc:824] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:42.914680 5505 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: No bootstrap required, opened a new log
I20260927 18:32:42.914939 5505 ts_tablet_manager.cc:1397] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 18:32:42.917585 5506 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: No bootstrap required, opened a new log
I20260927 18:32:42.917878 5506 ts_tablet_manager.cc:1397] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260927 18:32:42.919708 5508 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Bootstrap starting.
I20260927 18:32:42.922767 5508 tablet_bootstrap.cc:654] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:42.923702 5508 log.cc:824] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:42.925364 5505 raft_consensus.cc:348] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.925863 5505 raft_consensus.cc:374] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:42.925966 5505 raft_consensus.cc:729] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b4644d513247a7bdcdc601d7d1211b, State: Initialized, Role: FOLLOWER
I20260927 18:32:42.926515 5508 tablet_bootstrap.cc:492] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: No bootstrap required, opened a new log
I20260927 18:32:42.926759 5508 ts_tablet_manager.cc:1397] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 18:32:42.926607 5505 consensus_queue.cc:260] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.927630 5357 heartbeater.cc:503] Master 127.0.97.126:43827 was elected leader, sending a full tablet report...
I20260927 18:32:42.928278 5505 ts_tablet_manager.cc:1428] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 18:32:42.929325 5506 raft_consensus.cc:348] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.929733 5506 raft_consensus.cc:374] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:42.929850 5506 raft_consensus.cc:729] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fd6b3289c54f309a8d4a6ebe2a7285, State: Initialized, Role: FOLLOWER
I20260927 18:32:42.930505 5506 consensus_queue.cc:260] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.931558 5226 heartbeater.cc:503] Master 127.0.97.126:43827 was elected leader, sending a full tablet report...
I20260927 18:32:42.932168 5506 ts_tablet_manager.cc:1428] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 18:32:42.935873 5508 raft_consensus.cc:348] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.936268 5508 raft_consensus.cc:374] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:42.936352 5508 raft_consensus.cc:729] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff45f2e844d454c8a804ece2ecb7021, State: Initialized, Role: FOLLOWER
I20260927 18:32:42.936764 5508 consensus_queue.cc:260] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.937605 5487 heartbeater.cc:503] Master 127.0.97.126:43827 was elected leader, sending a full tablet report...
I20260927 18:32:42.938150 5508 ts_tablet_manager.cc:1428] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 18:32:42.971028 5513 raft_consensus.cc:482] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:42.971300 5513 raft_consensus.cc:504] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.972990 5513 leader_election.cc:288] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fd6b3289c54f309a8d4a6ebe2a7285 (127.0.97.65:44657), c1b4644d513247a7bdcdc601d7d1211b (127.0.97.66:43785)
I20260927 18:32:42.975678 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.973270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43785 (local address 127.0.97.67:34179)
0927 18:32:42.973544 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.973563 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.973671 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.974047 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.974053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.974076 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.974806 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.974810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.974829 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.974881 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.975264 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.975271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.975278 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.975393 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.975446 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 18:32:42.975890 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.973450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34179 (local address 127.0.97.66:43785)
0927 18:32:42.973673 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.973683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.973710 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:42.973828 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.973837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.973896 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.974010 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.975055 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.975061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.975068 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.975088 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.975127 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.975144 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.975272 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.975512 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.975584 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:32:42.976241 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.973377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51039 (local address 127.0.97.65:44657)
0927 18:32:42.973535 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:42.973541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:42.974209 (+ 668us) server_negotiation.cc:407] Connection header received
0927 18:32:42.974381 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:42.974387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:42.974450 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:42.974552 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:42.975557 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:42.975563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:42.975568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:42.975586 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:42.975625 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:42.975641 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:42.975771 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:42.975922 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:32:42.976008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:42.976927 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:42.973326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.97.67:51039)
0927 18:32:42.974096 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:42.974117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:42.974219 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:42.974605 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:42.974609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:42.974635 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:42.975338 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:32:42.975342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:42.975360 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:32:42.975399 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:42.975759 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:42.975765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:42.975772 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:32:42.976453 (+ 681us) client_negotiation.cc:240] Negotiation successful
0927 18:32:42.976505 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":425,"threads_started":1}
I20260927 18:32:42.976835 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc03f18108e44a698f6717b258559b6" candidate_uuid: "9ff45f2e844d454c8a804ece2ecb7021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b4644d513247a7bdcdc601d7d1211b" is_pre_election: true
I20260927 18:32:42.977275 5312 raft_consensus.cc:2454] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff45f2e844d454c8a804ece2ecb7021 in term 0.
I20260927 18:32:42.977877 5378 leader_election.cc:302] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff45f2e844d454c8a804ece2ecb7021, c1b4644d513247a7bdcdc601d7d1211b; no voters:
I20260927 18:32:42.977917 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc03f18108e44a698f6717b258559b6" candidate_uuid: "9ff45f2e844d454c8a804ece2ecb7021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" is_pre_election: true
I20260927 18:32:42.978219 5513 raft_consensus.cc:2781] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:42.978355 5513 raft_consensus.cc:482] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:42.978467 5513 raft_consensus.cc:3037] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:42.978456 5181 raft_consensus.cc:2454] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff45f2e844d454c8a804ece2ecb7021 in term 0.
I20260927 18:32:42.980628 5513 raft_consensus.cc:504] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
I20260927 18:32:42.981215 5513 leader_election.cc:288] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [CANDIDATE]: Term 1 election: Requested vote from peers c7fd6b3289c54f309a8d4a6ebe2a7285 (127.0.97.65:44657), c1b4644d513247a7bdcdc601d7d1211b (127.0.97.66:43785)
I20260927 18:32:42.981575 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc03f18108e44a698f6717b258559b6" candidate_uuid: "9ff45f2e844d454c8a804ece2ecb7021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285"
I20260927 18:32:42.981606 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc03f18108e44a698f6717b258559b6" candidate_uuid: "9ff45f2e844d454c8a804ece2ecb7021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b4644d513247a7bdcdc601d7d1211b"
I20260927 18:32:42.981801 5181 raft_consensus.cc:3037] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:42.981799 5312 raft_consensus.cc:3037] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:42.985056 5312 raft_consensus.cc:2454] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff45f2e844d454c8a804ece2ecb7021 in term 1.
I20260927 18:32:42.985074 5181 raft_consensus.cc:2454] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff45f2e844d454c8a804ece2ecb7021 in term 1.
I20260927 18:32:42.985426 5378 leader_election.cc:302] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff45f2e844d454c8a804ece2ecb7021, c1b4644d513247a7bdcdc601d7d1211b; no voters:
I20260927 18:32:42.985863 5513 raft_consensus.cc:2781] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:42.986344 5513 raft_consensus.cc:686] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 1 LEADER]: Becoming Leader. State: Replica: 9ff45f2e844d454c8a804ece2ecb7021, State: Running, Role: LEADER
I20260927 18:32:42.986840 5513 consensus_queue.cc:237] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } }
W20260927 18:32:42.987049 5227 tablet.cc:2369] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:42.991449 5040 catalog_manager.cc:5495] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ff45f2e844d454c8a804ece2ecb7021 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "9ff45f2e844d454c8a804ece2ecb7021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ff45f2e844d454c8a804ece2ecb7021" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:43.020817 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.018964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.0.97.67:34539)
0927 18:32:43.019123 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:43.019127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:43.019294 (+ 167us) server_negotiation.cc:407] Connection header received
0927 18:32:43.019459 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:43.019465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:43.019549 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:43.019637 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:43.020398 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:43.020402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:43.020405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:43.020419 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:43.020450 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:43.020461 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:43.020561 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:43.020592 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:32:43.020635 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:32:43.028832 5181 raft_consensus.cc:1261] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 1 FOLLOWER]: Refusing update from remote peer 9ff45f2e844d454c8a804ece2ecb7021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:43.028832 5312 raft_consensus.cc:1261] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 1 FOLLOWER]: Refusing update from remote peer 9ff45f2e844d454c8a804ece2ecb7021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:43.029757 5515 consensus_queue.cc:1035] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b4644d513247a7bdcdc601d7d1211b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:43.030139 5513 consensus_queue.cc:1035] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fd6b3289c54f309a8d4a6ebe2a7285" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:43.039294 5518 mvcc.cc:205] Tried to move back new op lower bound from 7334027112557039616 to 7334027112398544896. Current Snapshot: MvccSnapshot[applied={T|T < 7334027112557039616}]
I20260927 18:32:43.040920 5520 mvcc.cc:205] Tried to move back new op lower bound from 7334027112557039616 to 7334027112398544896. Current Snapshot: MvccSnapshot[applied={T|T < 7334027112557039616}]
I20260927 18:32:43.057482 5040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:43.057731 5040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:43.061291 5040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a69af3a2203846aba9b30e418cd52181: Sending DeleteTablet for 3 replicas of tablet 6cc03f18108e44a698f6717b258559b6
I20260927 18:32:43.062572 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc03f18108e44a698f6717b258559b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:43 UTC) from {username='slave'} at 127.0.0.1:53614
I20260927 18:32:43.062634 5161 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc03f18108e44a698f6717b258559b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:43 UTC) from {username='slave'} at 127.0.0.1:36686
I20260927 18:32:43.063308 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc03f18108e44a698f6717b258559b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:43 UTC) from {username='slave'} at 127.0.0.1:46756
I20260927 18:32:43.064287 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5099
I20260927 18:32:43.077023 5530 tablet_replica.cc:330] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: stopping tablet replica
I20260927 18:32:43.077649 5530 raft_consensus.cc:2229] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:43.080188 5530 raft_consensus.cc:2258] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:43.080358 5529 tablet_replica.cc:330] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285: stopping tablet replica
I20260927 18:32:43.080708 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.065582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.0.97.65:44657)
0927 18:32:43.065767 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:43.065792 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:43.078273 (+ 12481us) server_negotiation.cc:407] Connection header received
0927 18:32:43.078496 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:43.078502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:43.078622 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:43.078740 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:43.079699 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:43.079704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:43.079708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:43.079727 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:43.079767 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:43.079782 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:43.079905 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:43.080315 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 18:32:43.080381 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:32:43.080771 5528 tablet_replica.cc:330] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: stopping tablet replica
I20260927 18:32:43.080825 5529 raft_consensus.cc:2229] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:43.081159 5528 raft_consensus.cc:2229] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:43.081334 5529 raft_consensus.cc:2258] T 6cc03f18108e44a698f6717b258559b6 P c7fd6b3289c54f309a8d4a6ebe2a7285 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:43.081646 5528 raft_consensus.cc:2258] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:43.081830 5221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:43.083940 5528 ts_tablet_manager.cc:1905] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:43.087340 5530 ts_tablet_manager.cc:1905] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:43.090854 5528 ts_tablet_manager.cc:1918] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:43.091017 5528 log.cc:1196] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/6cc03f18108e44a698f6717b258559b6
I20260927 18:32:43.091533 5528 ts_tablet_manager.cc:1939] T 6cc03f18108e44a698f6717b258559b6 P c1b4644d513247a7bdcdc601d7d1211b: Deleting consensus metadata
I20260927 18:32:43.093391 5027 catalog_manager.cc:4841] TS c1b4644d513247a7bdcdc601d7d1211b (127.0.97.66:43785): tablet 6cc03f18108e44a698f6717b258559b6 (table test-table [id=4d3f162a943d4a98a7323095cfbfb8bf]) successfully deleted
I20260927 18:32:43.095121 5530 ts_tablet_manager.cc:1918] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:43.095305 5530 log.cc:1196] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/6cc03f18108e44a698f6717b258559b6
I20260927 18:32:43.095883 5530 ts_tablet_manager.cc:1939] T 6cc03f18108e44a698f6717b258559b6 P 9ff45f2e844d454c8a804ece2ecb7021: Deleting consensus metadata
I20260927 18:32:43.097769 5025 catalog_manager.cc:4841] TS 9ff45f2e844d454c8a804ece2ecb7021 (127.0.97.67:34539): tablet 6cc03f18108e44a698f6717b258559b6 (table test-table [id=4d3f162a943d4a98a7323095cfbfb8bf]) successfully deleted
I20260927 18:32:43.172907 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5099
I20260927 18:32:43.190618 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5230
W20260927 18:32:43.190575 5025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:44657 (error 108)
I20260927 18:32:43.192708 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.192217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36702)
0927 18:32:43.192408 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.192557 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:32:43.193001 5025 catalog_manager.cc:4584] TS c7fd6b3289c54f309a8d4a6ebe2a7285 (127.0.97.65:44657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cc03f18108e44a698f6717b258559b6: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
I20260927 18:32:43.194065 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.191531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.97.66:43785)
0927 18:32:43.191730 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:43.191737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:43.191960 (+ 223us) server_negotiation.cc:407] Connection header received
0927 18:32:43.192184 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:43.192190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:43.192309 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:43.192410 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:43.193401 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:43.193405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:43.193408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:43.193422 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:43.193451 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:43.193469 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:43.193598 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:43.193696 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:32:43.193741 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:32:43.194726 5352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:43.246794 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.246321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36704)
0927 18:32:43.246541 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.246626 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:32:43.247887 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.247545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36720)
0927 18:32:43.247693 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.247781 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:32:43.308753 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.308161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36732)
0927 18:32:43.308412 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.308557 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 18:32:43.309904 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.309494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36744)
0927 18:32:43.309653 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.309742 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:32:43.322518 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5230
I20260927 18:32:43.339010 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5360
I20260927 18:32:43.342041 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.339846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.0.97.67:34539)
0927 18:32:43.340053 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:43.340059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:43.340082 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:43.340285 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:43.340291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:43.340360 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:43.340469 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:43.341327 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:43.341333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:43.341338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:43.341357 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:43.341396 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:43.341410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:43.341559 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:43.341626 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 18:32:43.341685 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:32:43.342743 5482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:43.391158 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.390658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36754)
0927 18:32:43.390869 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.390990 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:32:43.392262 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.391835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36768)
0927 18:32:43.392040 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.392121 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 18:32:43.470317 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5360
I20260927 18:32:43.489432 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5009
I20260927 18:32:43.490716 5070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:43.597093 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:43.595733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44657 (local address 127.0.0.1:36780)
0927 18:32:43.596708 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:43.596832 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":819}
I20260927 18:32:43.597563 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5009
2026-09-27T18:32:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2648 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:32:43.638543 389 mini_oidc.cc:152] Starting JWKS server
I20260927 18:32:43.641039 389 webserver.cc:460] Webserver started at https://127.0.0.1:42311/ using document root <none> and password file <none>
I20260927 18:32:43.641357 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:32:43.642167 389 webserver.cc:460] Webserver started at http://127.0.0.1:42887/ using document root <none> and password file <none>
2026-09-27T18:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:43Z Disabled control of system clock
I20260927 18:32:43.667011 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35693
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:40575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/testchainca.cert with env {}
W20260927 18:32:43.977149 5542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:43.977417 5542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:43.977500 5542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:43.984475 5542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:43.984603 5542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:43.984650 5542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:43.984705 5542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:43.984747 5542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:32:43.984807 5542 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:32:43.984845 5542 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42311/jwks.json
I20260927 18:32:43.994535 5542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:43.995687 5542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:43.996851 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:44.007929 5550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:44.008026 5548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:44.008002 5547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:44.009128 5542 server_base.cc:1034] running on GCE node
I20260927 18:32:44.009932 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:44.011237 5542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:44.012420 5542 hybrid_clock.cc:648] HybridClock initialized: now 1790533964012375 us; error 42 us; skew 500 ppm
I20260927 18:32:44.012746 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:44.014923 5542 webserver.cc:460] Webserver started at http://127.0.97.126:41021/ using document root <none> and password file <none>
I20260927 18:32:44.015597 5542 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:44.015691 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:44.016001 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:44.018751 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "6418211a5eb0429dbb590957541719d1"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "f57756ad36681b11b6ebd33bcd9f3e98"
server_key_iv: "ed6b688fdbd642bf83f43a08f0890fb2"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.019501 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "6418211a5eb0429dbb590957541719d1"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "f57756ad36681b11b6ebd33bcd9f3e98"
server_key_iv: "ed6b688fdbd642bf83f43a08f0890fb2"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.024295 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 18:32:44.027562 5556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:44.028822 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:32:44.029000 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "6418211a5eb0429dbb590957541719d1"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "f57756ad36681b11b6ebd33bcd9f3e98"
server_key_iv: "ed6b688fdbd642bf83f43a08f0890fb2"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.029201 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:44.058646 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:44.059482 5542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:44.059767 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:44.077966 5542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35693
I20260927 18:32:44.077991 5607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35693 every 8 connection(s)
I20260927 18:32:44.079383 5542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:44.082876 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5542
I20260927 18:32:44.083105 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:44.083261 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:44.083794 389 external_mini_cluster.cc:1428] Setting key df5d7c871c42313b9cc1f911e7b514b2
I20260927 18:32:44.096015 5608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1: Bootstrap starting.
I20260927 18:32:44.097580 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:44.085555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.0.97.126:35693)
0927 18:32:44.086582 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:44.086592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:44.086631 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:32:44.088879 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:44.088899 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:44.089345 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:44.089694 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:44.090332 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.091535 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.093444 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.094144 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.094277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.095452 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:44.095459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:44.095471 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:44.095989 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:44.096060 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:44.096087 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:44.096209 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:44.096433 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:32:44.096712 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":386,"threads_started":1}
I20260927 18:32:44.099149 5608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:44.100277 5608 log.cc:824] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:44.101044 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:32:44.103477 5608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1: No bootstrap required, opened a new log
I20260927 18:32:44.110443 5608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } }
I20260927 18:32:44.110841 5608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:44.110957 5608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6418211a5eb0429dbb590957541719d1, State: Initialized, Role: FOLLOWER
I20260927 18:32:44.111509 5608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } }
I20260927 18:32:44.111752 5608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:44.111887 5608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:44.112051 5608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:44.114297 5608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } }
I20260927 18:32:44.114773 5608 leader_election.cc:302] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6418211a5eb0429dbb590957541719d1; no voters:
I20260927 18:32:44.115309 5608 leader_election.cc:288] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:44.115427 5613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:44.116148 5613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6418211a5eb0429dbb590957541719d1, State: Running, Role: LEADER
I20260927 18:32:44.116689 5613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } }
I20260927 18:32:44.117321 5608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:44.119445 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6418211a5eb0429dbb590957541719d1. Latest consensus state: current_term: 1 leader_uuid: "6418211a5eb0429dbb590957541719d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } } }
I20260927 18:32:44.119572 5614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6418211a5eb0429dbb590957541719d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6418211a5eb0429dbb590957541719d1" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35693 } } }
I20260927 18:32:44.119822 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:44.119867 5614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:44.120741 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:44.125728 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:44.133942 5620 catalog_manager.cc:1347] Generated new cluster ID: 0294a49ab0b84bfaa6d14029668ace4d
I20260927 18:32:44.134052 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:44.152839 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:44.154729 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:44.162077 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6418211a5eb0429dbb590957541719d1: Generated new TSK 0
I20260927 18:32:44.163292 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:32:44.200139 5638 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:32:44.200297 5638 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:32:44.201505 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:44.179568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.0.97.126:35693)
0927 18:32:44.179755 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:44.179762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:44.180213 (+ 451us) server_negotiation.cc:407] Connection header received
0927 18:32:44.180532 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:44.180538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:44.180631 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:44.180802 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:32:44.181243 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.182362 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.183757 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.184457 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.184597 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.185034 (+ 437us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:32:44.200766 (+ 15732us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:32:44.201140 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:32:44.201632 5611 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:32:44.202812 5637 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:44.179462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35693 (local address 127.0.0.1:48842)
0927 18:32:44.180118 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:44.180133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:44.180274 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:44.180811 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:44.180814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:44.180823 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:32:44.181087 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:44.181109 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.182524 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.182528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:44.183616 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:44.183628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.184604 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.184608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:44.184732 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.184783 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:32:44.201027 (+ 16244us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:32:44.201274 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:35693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":354,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.126:35693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:32:44.204466 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5542
I20260927 18:32:44.205519 5603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:44.295994 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5542
2026-09-27T18:32:44Z chronyd exiting
I20260927 18:32:44.345992 389 test_util.cc:175] -----------------------------------------------
I20260927 18:32:44.346118 389 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790533919012461-389-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (711 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:32:44.349942 389 mini_oidc.cc:152] Starting JWKS server
I20260927 18:32:44.351779 389 webserver.cc:460] Webserver started at https://127.0.0.1:36013/ using document root <none> and password file <none>
I20260927 18:32:44.351991 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:32:44.352594 389 webserver.cc:460] Webserver started at http://127.0.0.1:34743/ using document root <none> and password file <none>
2026-09-27T18:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:44Z Disabled control of system clock
I20260927 18:32:44.378309 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46229
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/testchainca.cert with env {}
W20260927 18:32:44.695760 5645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:44.696077 5645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:44.696230 5645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:44.703204 5645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:44.703347 5645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:44.703431 5645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:44.703527 5645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:44.703609 5645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:32:44.703698 5645 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:32:44.703764 5645 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36013/jwks.json
I20260927 18:32:44.713876 5645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:44.715178 5645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:44.716446 5645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:44.727782 5651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:44.727838 5650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:44.727856 5653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:44.729097 5645 server_base.cc:1034] running on GCE node
I20260927 18:32:44.729954 5645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:44.731339 5645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:44.732574 5645 hybrid_clock.cc:648] HybridClock initialized: now 1790533964732496 us; error 31 us; skew 500 ppm
I20260927 18:32:44.732867 5645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:44.738852 5645 webserver.cc:460] Webserver started at http://127.0.97.126:37861/ using document root <none> and password file <none>
I20260927 18:32:44.739490 5645 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:44.739586 5645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:44.739851 5645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:44.742650 5645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "12619dc6608f46b286a42c2f37ac483e"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "0fd7b7dced5d9c8aab4fe896abdae903"
server_key_iv: "a124d18077fce27003f1af15d37c8202"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.743342 5645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "12619dc6608f46b286a42c2f37ac483e"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "0fd7b7dced5d9c8aab4fe896abdae903"
server_key_iv: "a124d18077fce27003f1af15d37c8202"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.748307 5645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:44.751763 5659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:44.753216 5645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:44.753386 5645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "12619dc6608f46b286a42c2f37ac483e"
format_stamp: "Formatted at 2026-09-27 18:32:44 on dist-test-slave-n3p4"
server_key: "0fd7b7dced5d9c8aab4fe896abdae903"
server_key_iv: "a124d18077fce27003f1af15d37c8202"
server_key_version: "encryptionkey@0"
I20260927 18:32:44.753585 5645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:44.783320 5645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:44.784153 5645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:44.784416 5645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:44.803593 5645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46229
I20260927 18:32:44.803632 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46229 every 8 connection(s)
I20260927 18:32:44.804988 5645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:44.805519 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5645
I20260927 18:32:44.805840 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:44.806629 389 external_mini_cluster.cc:1428] Setting key 25fd9df6c777b6a08165c2bc81f0c329
I20260927 18:32:44.809427 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:44.820554 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:44.808503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.97.126:46229)
0927 18:32:44.809592 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:44.809610 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:44.809653 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:32:44.811395 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:44.811417 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:44.811908 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:44.812269 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:44.812837 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.814337 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.816163 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.816882 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.817019 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.818243 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:44.818250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:44.818265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:44.818860 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:44.818936 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:44.818957 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:44.819067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:44.819331 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:32:44.819687 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":404,"threads_started":1}
I20260927 18:32:44.821508 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e: Bootstrap starting.
I20260927 18:32:44.824410 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:32:44.824882 5711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:44.825855 5711 log.cc:824] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:44.828744 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e: No bootstrap required, opened a new log
I20260927 18:32:44.835798 5711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } }
I20260927 18:32:44.836136 5711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:44.836234 5711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12619dc6608f46b286a42c2f37ac483e, State: Initialized, Role: FOLLOWER
I20260927 18:32:44.836812 5711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } }
I20260927 18:32:44.837050 5711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:44.837177 5711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:44.837356 5711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:44.839413 5711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } }
I20260927 18:32:44.839886 5711 leader_election.cc:302] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12619dc6608f46b286a42c2f37ac483e; no voters:
I20260927 18:32:44.840401 5711 leader_election.cc:288] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:44.840587 5716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:44.841367 5716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [term 1 LEADER]: Becoming Leader. State: Replica: 12619dc6608f46b286a42c2f37ac483e, State: Running, Role: LEADER
I20260927 18:32:44.842010 5716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } }
I20260927 18:32:44.842464 5711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:44.845134 5718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12619dc6608f46b286a42c2f37ac483e. Latest consensus state: current_term: 1 leader_uuid: "12619dc6608f46b286a42c2f37ac483e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } } }
I20260927 18:32:44.845213 5717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12619dc6608f46b286a42c2f37ac483e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12619dc6608f46b286a42c2f37ac483e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46229 } } }
I20260927 18:32:44.845527 5718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:44.845577 5717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:44.846576 5724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:44.852648 5724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:44.861196 5724 catalog_manager.cc:1347] Generated new cluster ID: d3661e4ea0ca4b0391fc3027049d2dd6
I20260927 18:32:44.861320 5724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:44.880234 5724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:44.881551 5724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:44.893581 5724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12619dc6608f46b286a42c2f37ac483e: Generated new TSK 0
I20260927 18:32:44.894420 5724 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:32:44.948637 5741 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:32:44.948774 5741 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:32:44.949947 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:44.918798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.126:46229)
0927 18:32:44.919068 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:44.919077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:44.919546 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:32:44.919747 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:44.919752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:44.919841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:44.919992 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:32:44.920610 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.921549 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.923877 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.924377 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.924508 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.925106 (+ 598us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:32:44.949240 (+ 24134us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:32:44.949616 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 18:32:44.950109 5714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 18:32:44.950224 5740 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:44.918662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46229 (local address 127.0.0.1:57488)
0927 18:32:44.919372 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:44.919398 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:44.919557 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:44.920046 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:44.920052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:44.920067 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:32:44.920419 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:44.920431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.921718 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.921725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:44.923695 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:44.923711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:44.924571 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:44.924578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:44.924800 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:44.924847 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:32:44.949474 (+ 24627us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:32:44.949763 (+ 289us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:46229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":366,"threads_started":1}
I20260927 18:32:44.952241 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5645
I20260927 18:32:44.953469 5706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:45.053758 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5645
2026-09-27T18:32:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:32:45.108757 389 mini_oidc.cc:152] Starting JWKS server
I20260927 18:32:45.110602 389 webserver.cc:460] Webserver started at https://127.0.0.1:45343/ using document root <none> and password file <none>
I20260927 18:32:45.110826 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:32:45.111439 389 webserver.cc:460] Webserver started at http://127.0.0.1:32851/ using document root <none> and password file <none>
2026-09-27T18:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:45Z Disabled control of system clock
I20260927 18:32:45.137034 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33941
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45343/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:32:45.448503 5748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:45.448783 5748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:45.448931 5748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:45.456136 5748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:45.456238 5748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:45.456300 5748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:45.456401 5748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:45.456483 5748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:32:45.456584 5748 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:32:45.456657 5748 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45343/jwks.json
I20260927 18:32:45.465730 5748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45343/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:45.467002 5748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:45.468788 5748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:45.480613 5754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:45.480651 5753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:45.480830 5756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:45.481536 5748 server_base.cc:1034] running on GCE node
I20260927 18:32:45.482442 5748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:45.483834 5748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:45.485067 5748 hybrid_clock.cc:648] HybridClock initialized: now 1790533965485026 us; error 44 us; skew 500 ppm
I20260927 18:32:45.485411 5748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:45.487847 5748 webserver.cc:460] Webserver started at http://127.0.97.126:42609/ using document root <none> and password file <none>
I20260927 18:32:45.488605 5748 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:45.488735 5748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:45.489063 5748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:45.491933 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "e9e128f20b4b467d9e912bb8fe326f31"
format_stamp: "Formatted at 2026-09-27 18:32:45 on dist-test-slave-n3p4"
server_key: "16e7e17c762a3d8d2e5884f740ec430d"
server_key_iv: "3bf4ae145cada3294696a83e385e3872"
server_key_version: "encryptionkey@0"
I20260927 18:32:45.492781 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "e9e128f20b4b467d9e912bb8fe326f31"
format_stamp: "Formatted at 2026-09-27 18:32:45 on dist-test-slave-n3p4"
server_key: "16e7e17c762a3d8d2e5884f740ec430d"
server_key_iv: "3bf4ae145cada3294696a83e385e3872"
server_key_version: "encryptionkey@0"
I20260927 18:32:45.498045 5748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:32:45.501417 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:45.502904 5748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:45.503094 5748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "e9e128f20b4b467d9e912bb8fe326f31"
format_stamp: "Formatted at 2026-09-27 18:32:45 on dist-test-slave-n3p4"
server_key: "16e7e17c762a3d8d2e5884f740ec430d"
server_key_iv: "3bf4ae145cada3294696a83e385e3872"
server_key_version: "encryptionkey@0"
I20260927 18:32:45.503314 5748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:45.515957 5748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:45.516779 5748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:45.517059 5748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:45.535175 5748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33941
I20260927 18:32:45.535212 5813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33941 every 8 connection(s)
I20260927 18:32:45.536484 5748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:45.540220 5814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:45.543056 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5748
I20260927 18:32:45.543247 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:45.543876 389 external_mini_cluster.cc:1428] Setting key 3ccdcb565c0017a70472aedd6ac66927
I20260927 18:32:45.552273 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31: Bootstrap starting.
I20260927 18:32:45.555881 5814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:45.556928 5814 log.cc:824] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:45.557119 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:45.545706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.97.126:33941)
0927 18:32:45.546703 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:45.546714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:45.546753 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:32:45.548339 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:45.548364 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:45.548823 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:45.549139 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:45.549723 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.551063 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.552833 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.553465 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.553591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:45.554823 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:45.554829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:45.554843 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:45.555351 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:45.555433 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:45.555452 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:45.555534 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:45.555764 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:32:45.556144 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":346,"threads_started":1}
I20260927 18:32:45.559844 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31: No bootstrap required, opened a new log
I20260927 18:32:45.561194 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:32:45.568173 5814 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } }
I20260927 18:32:45.568572 5814 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:45.568683 5814 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e128f20b4b467d9e912bb8fe326f31, State: Initialized, Role: FOLLOWER
I20260927 18:32:45.569226 5814 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } }
I20260927 18:32:45.569460 5814 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:45.569612 5814 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:45.569772 5814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:45.571992 5814 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } }
I20260927 18:32:45.572508 5814 leader_election.cc:302] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9e128f20b4b467d9e912bb8fe326f31; no voters:
I20260927 18:32:45.573050 5814 leader_election.cc:288] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:45.573186 5819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:45.573935 5819 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [term 1 LEADER]: Becoming Leader. State: Replica: e9e128f20b4b467d9e912bb8fe326f31, State: Running, Role: LEADER
I20260927 18:32:45.574466 5819 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } }
I20260927 18:32:45.575084 5814 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:45.577178 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9e128f20b4b467d9e912bb8fe326f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } } }
I20260927 18:32:45.577245 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9e128f20b4b467d9e912bb8fe326f31. Latest consensus state: current_term: 1 leader_uuid: "e9e128f20b4b467d9e912bb8fe326f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e128f20b4b467d9e912bb8fe326f31" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33941 } } }
I20260927 18:32:45.577558 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:45.577577 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:45.579792 5826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:45.584408 5826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:45.592897 5826 catalog_manager.cc:1347] Generated new cluster ID: 764127c155b64414b4f6a05ec97e8a43
I20260927 18:32:45.593008 5826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:45.602254 5826 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:45.603555 5826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:45.613705 5826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9e128f20b4b467d9e912bb8fe326f31: Generated new TSK 0
I20260927 18:32:45.614553 5826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:32:45.642100 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:45.632859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.126:33941)
0927 18:32:45.633064 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:45.633071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:45.634353 (+ 1282us) server_negotiation.cc:407] Connection header received
0927 18:32:45.634551 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:45.634557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:45.634652 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:45.634833 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:32:45.635223 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.636323 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.637708 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.638404 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.638531 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:45.639080 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:32:45.641433 (+ 2353us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:32:45.641758 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:32:45.642223 5817 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:32:45.643783 5843 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:45.632736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33941 (local address 127.0.0.1:59314)
0927 18:32:45.634253 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:45.634269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:45.634389 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:45.634834 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:45.634838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:45.634846 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:32:45.635091 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:45.635098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.636475 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.636478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:45.637572 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:45.637583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:45.638639 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:45.638644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:45.638773 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:45.638806 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:32:45.641637 (+ 2831us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:32:45.643353 (+ 1716us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:33941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1377,"mutex_wait_us":1444,"thread_start_us":1232,"threads_started":1}
I20260927 18:32:45.645028 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5748
I20260927 18:32:45.646250 5809 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 18:32:45.654034 5844 webserver.cc:533] Webserver: sslize failed:
E20260927 18:32:45.654162 5844 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:32:45.735565 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5748
2026-09-27T18:32:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (681 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:45Z Disabled control of system clock
I20260927 18:32:45.815600 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37303
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37303
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/testchainca.cert with env {}
W20260927 18:32:46.127192 5849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:46.127485 5849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:46.127573 5849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:46.134553 5849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:46.134649 5849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:46.134686 5849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:46.134735 5849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:46.134778 5849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:32:46.134831 5849 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:32:46.134872 5849 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:32:46.144305 5849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:46.145502 5849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:46.146728 5849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:46.158064 5854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:46.158005 5855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:46.158221 5857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:46.159596 5849 server_base.cc:1034] running on GCE node
I20260927 18:32:46.160420 5849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:46.161948 5849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:46.163200 5849 hybrid_clock.cc:648] HybridClock initialized: now 1790533966163141 us; error 49 us; skew 500 ppm
I20260927 18:32:46.163589 5849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:32:46.165717 5849 webserver.cc:460] Webserver started at http://127.0.97.126:37615/ using document root <none> and password file <none>
I20260927 18:32:46.166409 5849 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:46.166580 5849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:46.166890 5849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:46.170190 5849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "cea3bae5e3e8443aa5df5dd525ae6a10"
format_stamp: "Formatted at 2026-09-27 18:32:46 on dist-test-slave-n3p4"
server_key: "711e7ccd367d204a440ec855af66c25e"
server_key_iv: "93f328ca8dda8c33f6b02c1c35612ba5"
server_key_version: "encryptionkey@0"
I20260927 18:32:46.171176 5849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "cea3bae5e3e8443aa5df5dd525ae6a10"
format_stamp: "Formatted at 2026-09-27 18:32:46 on dist-test-slave-n3p4"
server_key: "711e7ccd367d204a440ec855af66c25e"
server_key_iv: "93f328ca8dda8c33f6b02c1c35612ba5"
server_key_version: "encryptionkey@0"
I20260927 18:32:46.176433 5849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:32:46.180071 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:46.181391 5849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:32:46.181618 5849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "cea3bae5e3e8443aa5df5dd525ae6a10"
format_stamp: "Formatted at 2026-09-27 18:32:46 on dist-test-slave-n3p4"
server_key: "711e7ccd367d204a440ec855af66c25e"
server_key_iv: "93f328ca8dda8c33f6b02c1c35612ba5"
server_key_version: "encryptionkey@0"
I20260927 18:32:46.181870 5849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:46.203007 5849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:46.203822 5849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:46.204095 5849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:46.222198 5849 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37303
I20260927 18:32:46.222236 5914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37303 every 8 connection(s)
I20260927 18:32:46.223496 5849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:46.227365 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:46.231974 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5849
I20260927 18:32:46.232192 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:46.232791 389 external_mini_cluster.cc:1428] Setting key 5b3456e71c570a606e24e27f854ce874
I20260927 18:32:46.238893 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10: Bootstrap starting.
I20260927 18:32:46.242161 5915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:46.243350 5915 log.cc:824] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:46.246373 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:46.234681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.97.126:37303)
0927 18:32:46.235773 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:46.235783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:46.235825 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:32:46.237571 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:46.237593 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:46.238105 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:46.238464 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:46.239020 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.240289 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.241533 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.242285 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.242469 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:46.243871 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:46.243887 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:46.243907 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:32:46.244673 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:46.244770 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:32:46.244791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:46.244910 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:46.245122 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 18:32:46.245409 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":416,"threads_started":1}
I20260927 18:32:46.246447 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10: No bootstrap required, opened a new log
I20260927 18:32:46.250195 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:32:46.254370 5915 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } }
I20260927 18:32:46.254727 5915 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:46.254832 5915 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea3bae5e3e8443aa5df5dd525ae6a10, State: Initialized, Role: FOLLOWER
I20260927 18:32:46.255398 5915 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } }
I20260927 18:32:46.255632 5915 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:46.255766 5915 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:46.255934 5915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:46.258339 5915 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } }
I20260927 18:32:46.258802 5915 leader_election.cc:302] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea3bae5e3e8443aa5df5dd525ae6a10; no voters:
I20260927 18:32:46.259323 5915 leader_election.cc:288] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:46.259503 5920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:46.260218 5920 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [term 1 LEADER]: Becoming Leader. State: Replica: cea3bae5e3e8443aa5df5dd525ae6a10, State: Running, Role: LEADER
I20260927 18:32:46.260740 5920 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } }
I20260927 18:32:46.261360 5915 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:46.263674 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea3bae5e3e8443aa5df5dd525ae6a10. Latest consensus state: current_term: 1 leader_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } } }
I20260927 18:32:46.264081 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:46.264474 5921 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea3bae5e3e8443aa5df5dd525ae6a10" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37303 } } }
I20260927 18:32:46.264760 5921 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10 [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:46.265582 5929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:46.270758 5929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:46.279074 5929 catalog_manager.cc:1347] Generated new cluster ID: d6d26181a9ab43bbb187b98c7f27bc2e
I20260927 18:32:46.279170 5929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:46.316483 5929 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:46.317802 5929 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:46.330430 5929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea3bae5e3e8443aa5df5dd525ae6a10: Generated new TSK 0
I20260927 18:32:46.331262 5929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:32:46.418075 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:46.346998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.97.126:37303)
0927 18:32:46.347174 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:46.347178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:46.347630 (+ 452us) server_negotiation.cc:407] Connection header received
0927 18:32:46.347877 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:46.347881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:46.347942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:46.348102 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:32:46.348513 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.349234 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.350682 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.351140 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.351261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:46.351711 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:32:46.417300 (+ 65589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:32:46.417708 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57834: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:32:46.418260 5918 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57834: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 18:32:46.418337 5944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:46.347017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37303 (local address 127.0.0.1:57834)
0927 18:32:46.347529 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:46.347552 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:46.347703 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:46.348105 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:46.348110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:46.348121 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:32:46.348372 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:46.348382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.349372 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.349379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:46.350550 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:46.350562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:46.351307 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:46.351314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:46.351456 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:46.351486 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:32:46.417579 (+ 66093us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:32:46.417920 (+ 341us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:37303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260927 18:32:46.419759 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5849
I20260927 18:32:46.420814 5910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:46.520447 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5849
2026-09-27T18:32:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (764 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (412 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:47 dist-test-slave-n3p4 krb5kdc[5959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:47 dist-test-slave-n3p4 krb5kdc[5959](info): set up 2 sockets
Sep 27 18:32:47 dist-test-slave-n3p4 krb5kdc[5959](info): commencing operation
krb5kdc: starting...
W20260927 18:32:48.434600 389 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:48 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:32:48.630060 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37779
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:48.947607 5972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:48.947878 5972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:48.948025 5972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:48.955061 5972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:48.955156 5972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:48.955253 5972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:48.955343 5972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:48.955420 5972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:48.965278 5972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:48.966617 5972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:48.967887 5972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:48.979825 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:48.980229 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:48.979871 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:48.980602 5972 server_base.cc:1034] running on GCE node
I20260927 18:32:48.981397 5972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:48.982740 5972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:48.983964 5972 hybrid_clock.cc:648] HybridClock initialized: now 1790533968983923 us; error 44 us; skew 500 ppm
I20260927 18:32:48.984298 5972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:48 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:48.987283 5972 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:32:48.988574 5972 webserver.cc:460] Webserver started at http://127.0.97.126:37533/ using document root <none> and password file <none>
I20260927 18:32:48.989286 5972 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:48.989410 5972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:48.989837 5972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:48.994145 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "36c9d596764745d9abd123aca8dc231b"
format_stamp: "Formatted at 2026-09-27 18:32:48 on dist-test-slave-n3p4"
server_key: "d6994b9cbfc29dfd0fd746b34a709588"
server_key_iv: "eb508c51797e20e74bdaa022acba7e32"
server_key_version: "encryptionkey@0"
I20260927 18:32:48.995209 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "36c9d596764745d9abd123aca8dc231b"
format_stamp: "Formatted at 2026-09-27 18:32:48 on dist-test-slave-n3p4"
server_key: "d6994b9cbfc29dfd0fd746b34a709588"
server_key_iv: "eb508c51797e20e74bdaa022acba7e32"
server_key_version: "encryptionkey@0"
I20260927 18:32:49.002022 5972 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:32:49.005541 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:49.006844 5972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:49.007009 5972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "36c9d596764745d9abd123aca8dc231b"
format_stamp: "Formatted at 2026-09-27 18:32:48 on dist-test-slave-n3p4"
server_key: "d6994b9cbfc29dfd0fd746b34a709588"
server_key_iv: "eb508c51797e20e74bdaa022acba7e32"
server_key_version: "encryptionkey@0"
I20260927 18:32:49.007196 5972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:49.022759 5972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:49.026516 5972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:49.026769 5972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:49.046101 5972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37779
I20260927 18:32:49.046130 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37779 every 8 connection(s)
I20260927 18:32:49.047475 5972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:49.051618 6039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:49.057530 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 5972
I20260927 18:32:49.057756 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:49.058400 389 external_mini_cluster.cc:1428] Setting key fcb361b695e8b7d725fd6c99605abfa2
I20260927 18:32:49.063442 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b: Bootstrap starting.
I20260927 18:32:49.067603 6039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:49.068884 6039 log.cc:824] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:49.072078 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b: No bootstrap required, opened a new log
Sep 27 18:32:49 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:49.082398 6039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } }
I20260927 18:32:49.082852 6039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:49.082965 6039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c9d596764745d9abd123aca8dc231b, State: Initialized, Role: FOLLOWER
I20260927 18:32:49.083670 6039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } }
I20260927 18:32:49.083968 6039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:49.084120 6039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:49.084232 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:49.060568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.126:37779)
0927 18:32:49.061377 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:49.061394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:49.061446 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:32:49.063980 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:49.064019 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:49.064512 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:49.064946 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:49.066252 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.068404 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.070004 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.071048 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.071241 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:49.075166 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:49.075176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:49.075194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:49.075238 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:49.079509 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:49.080218 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:49.080233 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:49.080243 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:49.080483 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:49.080915 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:49.080925 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:49.080929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:49.081656 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:49.081991 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:49.082388 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 18:32:49.082819 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260927 18:32:49.084338 6039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:49.086975 6039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } }
I20260927 18:32:49.087471 6039 leader_election.cc:302] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c9d596764745d9abd123aca8dc231b; no voters:
I20260927 18:32:49.088001 6039 leader_election.cc:288] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:49.088140 6044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:49.088864 6044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [term 1 LEADER]: Becoming Leader. State: Replica: 36c9d596764745d9abd123aca8dc231b, State: Running, Role: LEADER
I20260927 18:32:49.089398 6044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } }
I20260927 18:32:49.090386 6039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:49.091944 6046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c9d596764745d9abd123aca8dc231b. Latest consensus state: current_term: 1 leader_uuid: "36c9d596764745d9abd123aca8dc231b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } } }
I20260927 18:32:49.092550 6045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c9d596764745d9abd123aca8dc231b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c9d596764745d9abd123aca8dc231b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37779 } } }
I20260927 18:32:49.092839 6046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:49.093433 6045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:49.097287 6053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:49.102166 6053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:49.111217 6053 catalog_manager.cc:1347] Generated new cluster ID: c5bd2dc4fdd44a438241835202fe73df
I20260927 18:32:49.111327 6053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:49.129700 6053 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:49.130977 6053 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:49.143527 6053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c9d596764745d9abd123aca8dc231b: Generated new TSK 0
I20260927 18:32:49.144326 6053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:32:49.201330 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:49.524928 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:49.525184 6065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:49.525332 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:49.532505 6065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:49.532619 6065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:49.532799 6065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:49.542102 6065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:49.543289 6065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:49.544521 6065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:49.556932 6071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:49.557068 6070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:49.557085 6073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:49.558056 6065 server_base.cc:1034] running on GCE node
I20260927 18:32:49.558614 6065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:49.559535 6065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:49.560726 6065 hybrid_clock.cc:648] HybridClock initialized: now 1790533969560675 us; error 44 us; skew 500 ppm
I20260927 18:32:49.561028 6065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:49 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:49.564085 6065 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:32:49.565423 6065 webserver.cc:460] Webserver started at http://127.0.97.65:46129/ using document root <none> and password file <none>
I20260927 18:32:49.566131 6065 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:49.566274 6065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:49.566560 6065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:49.569397 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "03046a5ddac046dfb055c355ee551382"
format_stamp: "Formatted at 2026-09-27 18:32:49 on dist-test-slave-n3p4"
server_key: "77ed074fa13aa9f5acedcbdc2ddb3971"
server_key_iv: "1b4b0707b898cc42a8d73d4eb942ab56"
server_key_version: "encryptionkey@0"
I20260927 18:32:49.570247 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "03046a5ddac046dfb055c355ee551382"
format_stamp: "Formatted at 2026-09-27 18:32:49 on dist-test-slave-n3p4"
server_key: "77ed074fa13aa9f5acedcbdc2ddb3971"
server_key_iv: "1b4b0707b898cc42a8d73d4eb942ab56"
server_key_version: "encryptionkey@0"
I20260927 18:32:49.575143 6065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:32:49.578445 6080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:49.579871 6065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:32:49.580067 6065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "03046a5ddac046dfb055c355ee551382"
format_stamp: "Formatted at 2026-09-27 18:32:49 on dist-test-slave-n3p4"
server_key: "77ed074fa13aa9f5acedcbdc2ddb3971"
server_key_iv: "1b4b0707b898cc42a8d73d4eb942ab56"
server_key_version: "encryptionkey@0"
I20260927 18:32:49.580291 6065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:49.598628 6065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:49.602882 6065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:49.603209 6065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:49.604194 6065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:49.605659 6065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:49.605764 6065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:49.605952 6065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:49.606050 6065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:49.641708 6065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37801
I20260927 18:32:49.641757 6192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37801 every 8 connection(s)
I20260927 18:32:49.643096 6065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:49.651157 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6065
I20260927 18:32:49.651376 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:49.651995 389 external_mini_cluster.cc:1428] Setting key 5dc72d658b1083df86c7e1f607f1135b
Sep 27 18:32:49 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:49.663213 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:49.645634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48359 (local address 127.0.97.126:37779)
0927 18:32:49.646052 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:49.646056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:49.647001 (+ 945us) server_negotiation.cc:407] Connection header received
0927 18:32:49.648727 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:49.648732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:49.648824 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:49.648937 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:49.650393 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.651113 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.652802 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.653306 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.653450 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:49.657128 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:49.657133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:49.657136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:49.657166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:49.659732 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:49.660494 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:49.660502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:49.660509 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:49.660656 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:49.661048 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:49.661056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:49.661062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:49.661424 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:49.661574 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:49.662429 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 18:32:49.662702 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":219,"threads_started":1}
I20260927 18:32:49.663753 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:49.646136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37779 (local address 127.0.97.65:48359)
0927 18:32:49.646834 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:49.646873 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:49.648476 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:49.649150 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:49.649159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:49.649632 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:49.650229 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:49.650245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.651282 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.651289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:49.652645 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:49.652657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:49.653581 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:49.653590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:49.653865 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:49.654515 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:49.654541 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:49.656871 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:49.659861 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:49.659868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:49.659880 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:49.660341 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:49.660763 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:49.660766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:49.660768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:49.660937 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:49.661590 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:49.661596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:49.662031 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 18:32:49.662265 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 18:32:49.662657 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":303,"threads_started":1}
I20260927 18:32:49.665266 6193 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37779
I20260927 18:32:49.665576 6193 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:49.666352 6193 heartbeater.cc:511] Master 127.0.97.126:37779 requested a full tablet report, sending...
I20260927 18:32:49.668447 6004 ts_manager.cc:194] Registered new tserver with Master: 03046a5ddac046dfb055c355ee551382 (127.0.97.65:37801)
I20260927 18:32:49.670006 6004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:48359
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:32:49.697922 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:50.014343 6199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:50.014668 6199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:50.014850 6199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:50.022603 6199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:50.022753 6199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:50.022956 6199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:50.033517 6199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:50.034930 6199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:50.036387 6199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:50.050549 6204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:50.050608 6205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:50.051399 6207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:50.052276 6199 server_base.cc:1034] running on GCE node
I20260927 18:32:50.053030 6199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:50.054296 6199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:50.055588 6199 hybrid_clock.cc:648] HybridClock initialized: now 1790533970055479 us; error 103 us; skew 500 ppm
I20260927 18:32:50.055950 6199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.063603 6199 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:32:50.065352 6199 webserver.cc:460] Webserver started at http://127.0.97.66:36545/ using document root <none> and password file <none>
I20260927 18:32:50.066207 6199 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:50.066358 6199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:50.066669 6199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:50.069628 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "075588a09ee045689841908629e5266d"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "e82a3a4c280c114e280fdd303f6bbe7b"
server_key_iv: "6d796965a1c7804e339417132236ee6f"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.070523 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "075588a09ee045689841908629e5266d"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "e82a3a4c280c114e280fdd303f6bbe7b"
server_key_iv: "6d796965a1c7804e339417132236ee6f"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.075950 6199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:50.079532 6214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.080870 6199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:32:50.081091 6199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "075588a09ee045689841908629e5266d"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "e82a3a4c280c114e280fdd303f6bbe7b"
server_key_iv: "6d796965a1c7804e339417132236ee6f"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.081287 6199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:50.096249 6199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:50.100292 6199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:50.100569 6199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:50.101486 6199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:50.102813 6199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:50.102908 6199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.103044 6199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:50.103117 6199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.135082 6199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:34057
I20260927 18:32:50.135116 6326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:34057 every 8 connection(s)
I20260927 18:32:50.136307 6199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:50.145916 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6199
I20260927 18:32:50.146111 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:50.146621 389 external_mini_cluster.cc:1428] Setting key c200106602263b640225f71a15419451
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:50.155869 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.138764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34527 (local address 127.0.97.126:37779)
0927 18:32:50.138961 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.138991 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.140191 (+ 1200us) server_negotiation.cc:407] Connection header received
0927 18:32:50.141901 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.141905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.141986 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.142074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.143444 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.144175 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.145834 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.146322 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.146415 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.150098 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.150103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.150107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.150135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.152945 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.153629 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.153637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.153640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.153770 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.154241 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.154246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.154249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.154509 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.154652 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.155317 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.155544 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:50.156795 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.139323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37779 (local address 127.0.97.66:34527)
0927 18:32:50.140034 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:50.140071 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:50.141604 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:50.142272 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:50.142280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:50.142766 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:50.143296 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:50.143314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.144298 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.144303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:50.145633 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:50.145649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.146526 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.146534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:50.146845 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.147595 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:50.147625 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:50.149895 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:50.153081 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:50.153090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:50.153105 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:50.153514 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:50.153925 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:50.153930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:50.153934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:50.154125 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:50.154639 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:50.154647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:50.154973 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 18:32:50.155230 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 18:32:50.155728 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":295,"threads_started":1}
I20260927 18:32:50.158334 6327 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37779
I20260927 18:32:50.158660 6327 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:50.159360 6327 heartbeater.cc:511] Master 127.0.97.126:37779 requested a full tablet report, sending...
I20260927 18:32:50.160800 6004 ts_manager.cc:194] Registered new tserver with Master: 075588a09ee045689841908629e5266d (127.0.97.66:34057)
I20260927 18:32:50.161614 6004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:34527
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:32:50.191107 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:50.509831 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:50.510175 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:50.510337 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:50.517730 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:50.517901 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:50.518115 6332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:50.527601 6332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:50.528858 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:50.530249 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:50.542440 6340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:50.542533 6337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:50.542533 6338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:50.543637 6332 server_base.cc:1034] running on GCE node
I20260927 18:32:50.544232 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:50.545145 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:50.546368 6332 hybrid_clock.cc:648] HybridClock initialized: now 1790533970546302 us; error 48 us; skew 500 ppm
I20260927 18:32:50.546744 6332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.550112 6332 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:32:50.551558 6332 webserver.cc:460] Webserver started at http://127.0.97.67:43049/ using document root <none> and password file <none>
I20260927 18:32:50.552235 6332 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:50.552368 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:50.552690 6332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:50.555668 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "14a6ea267f1f44e2a35770a09ba062df"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "b63d629a50a912a8f8134ef289d1c8d7"
server_key_iv: "149603f0fff09ce76f40c6333def513c"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.556461 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "14a6ea267f1f44e2a35770a09ba062df"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "b63d629a50a912a8f8134ef289d1c8d7"
server_key_iv: "149603f0fff09ce76f40c6333def513c"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.561568 6332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:32:50.565055 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.566424 6332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:50.566614 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "14a6ea267f1f44e2a35770a09ba062df"
format_stamp: "Formatted at 2026-09-27 18:32:50 on dist-test-slave-n3p4"
server_key: "b63d629a50a912a8f8134ef289d1c8d7"
server_key_iv: "149603f0fff09ce76f40c6333def513c"
server_key_version: "encryptionkey@0"
I20260927 18:32:50.566826 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:50.578351 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:50.582759 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:50.583065 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:50.584131 6332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:50.585604 6332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:50.585716 6332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.585878 6332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:50.585964 6332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:50.620440 6332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:34405
I20260927 18:32:50.620476 6459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:34405 every 8 connection(s)
I20260927 18:32:50.621688 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:50.630158 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6332
I20260927 18:32:50.630390 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:50.630908 389 external_mini_cluster.cc:1428] Setting key 9c1748b07a833882d23964d8a3fbe2fd
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:32:50.641059 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.624068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36341 (local address 127.0.97.126:37779)
0927 18:32:50.624286 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.624292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.625387 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 18:32:50.627154 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.627158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.627225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.627314 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.628770 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.629611 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.631350 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.631845 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.631969 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.635745 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.635750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.635754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.635787 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.638259 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.638896 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.638899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.638901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.639034 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.639546 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.639551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.639554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.639828 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.639933 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.640637 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.640802 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:32:50.642192 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.624540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37779 (local address 127.0.97.67:36341)
0927 18:32:50.625229 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:50.625272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:50.626893 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:50.627523 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:50.627531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:50.628044 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:50.628577 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:50.628593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.629741 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.629744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:50.631186 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:50.631197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.632106 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.632114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:50.632407 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.633195 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:50.633226 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:50.635556 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:50.638396 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:50.638402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:50.638414 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:50.638790 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:50.639123 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:50.639142 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:50.639145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:50.639315 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:50.639926 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:50.639931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:50.640232 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 18:32:50.640441 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 18:32:50.640801 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":284,"threads_started":1}
I20260927 18:32:50.643671 6460 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37779
I20260927 18:32:50.643976 6460 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:50.644817 6460 heartbeater.cc:511] Master 127.0.97.126:37779 requested a full tablet report, sending...
I20260927 18:32:50.646471 6004 ts_manager.cc:194] Registered new tserver with Master: 14a6ea267f1f44e2a35770a09ba062df (127.0.97.67:34405)
I20260927 18:32:50.647433 6004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:36341
I20260927 18:32:50.659425 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:32:50.672780 6193 heartbeater.cc:503] Master 127.0.97.126:37779 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/test-admin.keytab.
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.687304 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:32:50.692503 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:50.689983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.0.97.65:37801)
0927 18:32:50.690574 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.690581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.690608 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:32:50.690750 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.690756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.691055 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.691179 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.692073 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55752: BlockingRecv error: recv got EOF from 127.0.0.1:55752 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":287,"threads_started":1}
W20260927 18:32:50.692510 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.694166 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.695621 389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:50.710891 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.697448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.97.65:37801)
0927 18:32:50.697721 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.697726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.697872 (+ 146us) server_negotiation.cc:407] Connection header received
0927 18:32:50.698015 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.698018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.698071 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.698146 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.699135 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.700094 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.701337 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.701958 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.702049 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.705533 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.705544 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.705558 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.705590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.708369 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.708989 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.708995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.708998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.709106 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.709428 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.709436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.709439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.709834 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.710084 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.710353 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.710580 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
W20260927 18:32:50.716949 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:50.715152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.97.65:37801)
0927 18:32:50.715368 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.715374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.715587 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:32:50.715775 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.715780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.715860 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.715968 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.716693 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55760: BlockingRecv error: recv got EOF from 127.0.0.1:55760 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 18:32:50.717350 389 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc)
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.718902 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.720278 389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:50.733877 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.721508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.0.97.65:37801)
0927 18:32:50.721664 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.721672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.722069 (+ 397us) server_negotiation.cc:407] Connection header received
0927 18:32:50.722231 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.722236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.722299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.722391 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.723287 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.723988 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.725145 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.725577 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.725696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.729030 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.729036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.729039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.729077 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.731473 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.732124 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.732130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.732134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.732251 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.732601 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.732606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.732609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.732855 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.733110 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.733366 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.733565 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:32:50.738253 6467 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:32:50.736221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.0.97.65:37801)
0927 18:32:50.736361 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.736365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.736800 (+ 435us) server_negotiation.cc:407] Connection header received
0927 18:32:50.736981 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.736987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.737067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.737170 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.737989 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55780: BlockingRecv error: recv got EOF from 127.0.0.1:55780 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 18:32:50.738392 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790533919012461-389-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.739912 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:50.741309 389 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:32:50.756485 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.742730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.97.65:37801)
0927 18:32:50.742923 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.742931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.743137 (+ 206us) server_negotiation.cc:407] Connection header received
0927 18:32:50.743402 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.743415 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.743499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.743672 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.744677 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.745436 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.746882 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.747330 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.747467 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.751193 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.751199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.751203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.751235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.754189 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.754783 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.754787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.754789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.754933 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.755348 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.755352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.755354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.755647 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.755872 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.756085 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.756235 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:50.758069 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6065
I20260927 18:32:50.769318 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.758751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.0.97.65:37801)
0927 18:32:50.758884 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.758887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.759198 (+ 311us) server_negotiation.cc:407] Connection header received
0927 18:32:50.759311 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.759314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.759361 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.759434 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.760254 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.760901 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.762179 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.762616 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.762733 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.764609 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.764614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.764618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.764646 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.767259 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.767794 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.767798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.767800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.767917 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.768226 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.768229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.768231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.768510 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.768720 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.768896 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.769054 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:32:50.769929 6188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:50.853888 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6065
I20260927 18:32:50.870055 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6199
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:32:50.885008 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.871012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.97.66:34057)
0927 18:32:50.871498 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.871505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.871537 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:32:50.871596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.871602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.871816 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.871983 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.872915 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.874111 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.875306 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.875853 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.875983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.879733 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.879743 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.879757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.879789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.882434 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.883070 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.883084 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.883087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.883199 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:50.883552 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:50.883557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:50.883561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:50.883867 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:50.884190 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:50.884479 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:32:50.884720 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":237,"threads_started":1}
I20260927 18:32:50.886662 6322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:50.968539 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6199
I20260927 18:32:50.984349 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6332
Sep 27 18:32:50 dist-test-slave-n3p4 krb5kdc[5959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:32:51.002260 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:50.985360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.0.97.67:34405)
0927 18:32:50.986233 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:50.986245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:50.986292 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:32:50.986378 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:50.986391 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:50.986702 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:50.986857 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:50.988262 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.989278 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.990743 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:50.991858 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:50.992034 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:50.996483 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:50.996503 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:50.996525 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:50.996577 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:50.999802 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:51.000402 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:51.000409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:51.000411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:51.000551 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:51.000872 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:51.000876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:51.000878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:51.001219 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:51.001480 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:51.001645 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 18:32:51.001938 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":455,"threads_started":1}
I20260927 18:32:51.003533 6455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:51.085289 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6332
I20260927 18:32:51.100513 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 5972
I20260927 18:32:51.101635 6034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:51.202606 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 5972
2026-09-27T18:32:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3862 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:32:51 dist-test-slave-n3p4 krb5kdc[6487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:32:51 dist-test-slave-n3p4 krb5kdc[6487](info): set up 2 sockets
Sep 27 18:32:51 dist-test-slave-n3p4 krb5kdc[6487](info): commencing operation
krb5kdc: starting...
W20260927 18:32:52.290619 389 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:32:52 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:32:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260927 18:32:52.501572 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35643
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:52.825120 6500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:52.825455 6500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:52.825605 6500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:52.832810 6500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:52.832988 6500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:52.833089 6500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:52.833189 6500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:52.833271 6500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:52.843030 6500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:52.844343 6500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:52.845683 6500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:52.860306 6505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:52.860921 6508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:52.861028 6506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:52.862044 6500 server_base.cc:1034] running on GCE node
I20260927 18:32:52.862743 6500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:52.864111 6500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:52.865340 6500 hybrid_clock.cc:648] HybridClock initialized: now 1790533972865286 us; error 45 us; skew 500 ppm
I20260927 18:32:52.865728 6500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:52 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:52.868801 6500 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260927 18:32:52.870308 6500 webserver.cc:460] Webserver started at http://127.0.97.126:42147/ using document root <none> and password file <none>
I20260927 18:32:52.870966 6500 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:52.871101 6500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:52.871381 6500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:52.874328 6500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "c310d59190374de6940119e43dcbc32f"
format_stamp: "Formatted at 2026-09-27 18:32:52 on dist-test-slave-n3p4"
server_key: "1161e58611fea0b21a4407296f684672"
server_key_iv: "2740f25c3d4f515bdfd738f9be2ea712"
server_key_version: "encryptionkey@0"
I20260927 18:32:52.875097 6500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "c310d59190374de6940119e43dcbc32f"
format_stamp: "Formatted at 2026-09-27 18:32:52 on dist-test-slave-n3p4"
server_key: "1161e58611fea0b21a4407296f684672"
server_key_iv: "2740f25c3d4f515bdfd738f9be2ea712"
server_key_version: "encryptionkey@0"
I20260927 18:32:52.880244 6500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:52.883590 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:52.884938 6500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:32:52.885106 6500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "c310d59190374de6940119e43dcbc32f"
format_stamp: "Formatted at 2026-09-27 18:32:52 on dist-test-slave-n3p4"
server_key: "1161e58611fea0b21a4407296f684672"
server_key_iv: "2740f25c3d4f515bdfd738f9be2ea712"
server_key_version: "encryptionkey@0"
I20260927 18:32:52.885295 6500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:52.908396 6500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:52.912914 6500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:52.913192 6500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:52.932163 6500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35643
I20260927 18:32:52.932193 6566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35643 every 8 connection(s)
I20260927 18:32:52.933516 6500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:32:52.937173 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:52.941205 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6500
I20260927 18:32:52.941452 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:32:52.942304 389 external_mini_cluster.cc:1428] Setting key 3b4bcfac3bd48a98306e2d0345426c58
I20260927 18:32:52.949543 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Bootstrap starting.
I20260927 18:32:52.953004 6567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:52.954151 6567 log.cc:824] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:52.957559 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: No bootstrap required, opened a new log
Sep 27 18:32:52 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:52.965269 6567 raft_consensus.cc:348] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } }
I20260927 18:32:52.965669 6567 raft_consensus.cc:374] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:52.965799 6567 raft_consensus.cc:729] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c310d59190374de6940119e43dcbc32f, State: Initialized, Role: FOLLOWER
I20260927 18:32:52.966394 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:52.944437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.0.97.126:35643)
0927 18:32:52.945299 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:52.945311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:52.945350 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:32:52.947985 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:52.948007 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:52.948384 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:52.948699 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:52.950403 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:52.951690 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:52.953396 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:52.954136 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:52.954299 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:52.958859 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:52.958867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:52.958880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:52.958912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:52.962202 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:52.962895 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:52.962907 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:52.962913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:52.963075 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:52.963433 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:52.963439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:52.963443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:52.964007 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:52.964304 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:52.964716 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 18:32:52.965143 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":413,"threads_started":1}
I20260927 18:32:52.966409 6567 consensus_queue.cc:260] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } }
I20260927 18:32:52.966763 6567 raft_consensus.cc:388] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:32:52.966912 6567 raft_consensus.cc:482] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:32:52.967082 6567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:52.969501 6567 raft_consensus.cc:504] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } }
I20260927 18:32:52.970049 6567 leader_election.cc:302] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c310d59190374de6940119e43dcbc32f; no voters:
I20260927 18:32:52.970676 6567 leader_election.cc:288] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:32:52.970818 6572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:52.971508 6572 raft_consensus.cc:686] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 LEADER]: Becoming Leader. State: Replica: c310d59190374de6940119e43dcbc32f, State: Running, Role: LEADER
I20260927 18:32:52.972075 6572 consensus_queue.cc:237] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } }
I20260927 18:32:52.972782 6567 sys_catalog.cc:564] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:52.974993 6574 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c310d59190374de6940119e43dcbc32f. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } }
I20260927 18:32:52.974960 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } }
I20260927 18:32:52.975338 6574 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:52.975415 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:52.976334 6580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:52.982553 6580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:52.992959 6580 catalog_manager.cc:1347] Generated new cluster ID: b7516ade86d64b1f93f405d4f89955e4
I20260927 18:32:52.993083 6580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:53.014521 6580 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:32:53.015954 6580 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:53.021471 6580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Generated new TSK 0
I20260927 18:32:53.022580 6580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260927 18:32:53.073942 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:53.405011 6593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:53.405334 6593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:53.405501 6593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:53.413439 6593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:53.413594 6593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:53.413810 6593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:53.423746 6593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:53.425112 6593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:53.426476 6593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:53.439158 6598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:53.439246 6599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:53.439201 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:53.440088 6593 server_base.cc:1034] running on GCE node
I20260927 18:32:53.440737 6593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:53.441725 6593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:53.442965 6593 hybrid_clock.cc:648] HybridClock initialized: now 1790533973442904 us; error 65 us; skew 500 ppm
I20260927 18:32:53.443279 6593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:53 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:53.446475 6593 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260927 18:32:53.448004 6593 webserver.cc:460] Webserver started at http://127.0.97.65:34341/ using document root <none> and password file <none>
I20260927 18:32:53.448689 6593 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:53.448822 6593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:53.449106 6593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:53.452091 6593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "7365fcbd04ca4282b63a4c0908b1b5fd"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "97fdb080a83a5a6a644199b3581fe286"
server_key_iv: "0a73769b098a16f46f04a2b8ed780b8d"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.452869 6593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "7365fcbd04ca4282b63a4c0908b1b5fd"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "97fdb080a83a5a6a644199b3581fe286"
server_key_iv: "0a73769b098a16f46f04a2b8ed780b8d"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.457907 6593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:32:53.461259 6608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:53.462617 6593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:32:53.462785 6593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "7365fcbd04ca4282b63a4c0908b1b5fd"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "97fdb080a83a5a6a644199b3581fe286"
server_key_iv: "0a73769b098a16f46f04a2b8ed780b8d"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.462975 6593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:53.481395 6593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:53.485997 6593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:53.486285 6593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:53.487258 6593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:53.488663 6593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:53.488730 6593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:53.488862 6593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:53.488919 6593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:53.520202 6593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43871
I20260927 18:32:53.520241 6720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43871 every 8 connection(s)
I20260927 18:32:53.521519 6593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:53.526142 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6593
I20260927 18:32:53.526376 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:32:53.527179 389 external_mini_cluster.cc:1428] Setting key bdd79aaa821070404e6bb3997235c8ac
Sep 27 18:32:53 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:53.545356 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:53.524067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57527 (local address 127.0.97.126:35643)
0927 18:32:53.524495 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:53.524501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:53.525573 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 18:32:53.528233 (+ 2660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:53.528256 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:53.528347 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:53.528463 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:53.530608 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:53.531326 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:53.533833 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:53.534251 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:53.534354 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:53.538733 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:53.538738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:53.538742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:53.538774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:53.541539 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:53.542580 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:53.542585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:53.542588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:53.542730 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:53.543395 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:53.543399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:53.543401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:53.543725 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:53.543841 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:53.544837 (+ 996us) server_negotiation.cc:299] Negotiation successful
0927 18:32:53.545007 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260927 18:32:53.547049 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:53.524708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:57527)
0927 18:32:53.525393 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:53.525439 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:53.527909 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:53.528691 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:53.528700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:53.529436 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:53.530407 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:53.530435 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:53.531483 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:53.531492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:53.533634 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:53.533652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:53.534499 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:53.534508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:53.534842 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:53.535581 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:53.535621 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:53.538500 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:53.541694 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:53.541706 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:53.541726 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:53.542410 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:53.542853 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:53.542857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:53.542863 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:53.543112 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:53.543873 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:53.543886 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:53.544399 (+ 513us) client_negotiation.cc:769] Sending connection context
0927 18:32:53.544746 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 18:32:53.545374 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":271,"threads_started":1}
I20260927 18:32:53.548823 6721 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:53.549127 6721 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:53.549896 6721 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:32:53.551915 6532 ts_manager.cc:194] Registered new tserver with Master: 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
I20260927 18:32:53.553691 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:57527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260927 18:32:53.579463 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:53.913249 6727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:53.913543 6727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:53.913715 6727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:53.921348 6727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:53.921489 6727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:53.921667 6727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:53.931829 6727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:53.933353 6727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:53.934727 6727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:53.948249 6735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:53.948294 6733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:53.948385 6732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:53.949330 6727 server_base.cc:1034] running on GCE node
I20260927 18:32:53.950052 6727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:53.951032 6727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:53.952239 6727 hybrid_clock.cc:648] HybridClock initialized: now 1790533973952176 us; error 57 us; skew 500 ppm
I20260927 18:32:53.952607 6727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:53 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:53.955891 6727 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260927 18:32:53.957263 6727 webserver.cc:460] Webserver started at http://127.0.97.66:34345/ using document root <none> and password file <none>
I20260927 18:32:53.958032 6727 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:53.958175 6727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:53.958474 6727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:53.961398 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "dfae4e78bf9540788eb6ecd5dc311565"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "ea4e55f20589dbb23a08a0321d6684e8"
server_key_iv: "f6d6cc3b2b2caf390e06ff280eae1134"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.962188 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "dfae4e78bf9540788eb6ecd5dc311565"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "ea4e55f20589dbb23a08a0321d6684e8"
server_key_iv: "f6d6cc3b2b2caf390e06ff280eae1134"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.967424 6727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:32:53.970709 6742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:53.971951 6727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:32:53.972111 6727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "dfae4e78bf9540788eb6ecd5dc311565"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "ea4e55f20589dbb23a08a0321d6684e8"
server_key_iv: "f6d6cc3b2b2caf390e06ff280eae1134"
server_key_version: "encryptionkey@0"
I20260927 18:32:53.972292 6727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:53.984233 6727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:53.988478 6727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:53.988809 6727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:53.989756 6727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:53.991497 6727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:53.991608 6727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:53.991736 6727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:53.991811 6727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:54.025285 6727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39207
I20260927 18:32:54.025331 6854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39207 every 8 connection(s)
I20260927 18:32:54.026772 6727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:54.030174 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6727
I20260927 18:32:54.030383 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:32:54.030958 389 external_mini_cluster.cc:1428] Setting key c0647fd82fa3f19810228a18374caec2
Sep 27 18:32:54 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:54.048136 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:54.029423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55153 (local address 127.0.97.126:35643)
0927 18:32:54.029597 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:54.029601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:54.030961 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 18:32:54.032895 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:54.032900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:54.033000 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:54.033122 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:54.034602 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.035315 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.036838 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.037270 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.037404 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:54.041090 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:54.041099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:54.041107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:54.041151 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:54.044668 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:54.045414 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:54.045422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:54.045429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:54.045584 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:54.046279 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:54.046286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:54.046293 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:54.046667 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:54.046816 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:54.047437 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 18:32:54.047686 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:32:54.048995 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:54.030012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:55153)
0927 18:32:54.030820 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:54.030857 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:54.032659 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:54.033338 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:54.033345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:54.033881 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:54.034419 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:54.034436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.035471 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.035475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:54.036688 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:54.036704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.037509 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.037516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:54.037772 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:54.038547 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:54.038580 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:54.040854 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:54.044824 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:54.044831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:54.044843 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:54.045275 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:54.045710 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:54.045714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:54.045716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:54.046143 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:54.046791 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:54.046797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:54.047085 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 18:32:54.047296 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 18:32:54.047804 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":336,"threads_started":1}
I20260927 18:32:54.050566 6855 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:54.050907 6855 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:54.051767 6855 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:32:54.053416 6532 ts_manager.cc:194] Registered new tserver with Master: dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:54.054626 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:55153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260927 18:32:54.079262 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:54.397142 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:54.397452 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:54.397614 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:54.405025 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:54.405182 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:54.405368 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:54.415201 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:54.416498 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:54.417925 6860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:54.431536 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:54.431555 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:54.431596 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:54.432637 6860 server_base.cc:1034] running on GCE node
I20260927 18:32:54.433209 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:54.434167 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:54.435413 6860 hybrid_clock.cc:648] HybridClock initialized: now 1790533974435355 us; error 27 us; skew 500 ppm
I20260927 18:32:54.435840 6860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:54 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:54.439075 6860 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260927 18:32:54.440574 6860 webserver.cc:460] Webserver started at http://127.0.97.67:36061/ using document root <none> and password file <none>
I20260927 18:32:54.441277 6860 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:54.441422 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:54.441730 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:54.444607 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "4d273b5452b34e25b8d51b42f0271c26"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "41ec6c95633d94745fae419113dc7413"
server_key_iv: "9812db775b904cf4d4700c2fa73f4b93"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.445364 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "4d273b5452b34e25b8d51b42f0271c26"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "41ec6c95633d94745fae419113dc7413"
server_key_iv: "9812db775b904cf4d4700c2fa73f4b93"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.450644 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:32:54.454131 6875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:54.455442 6860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:32:54.455642 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "4d273b5452b34e25b8d51b42f0271c26"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "41ec6c95633d94745fae419113dc7413"
server_key_iv: "9812db775b904cf4d4700c2fa73f4b93"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.455842 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:54.488480 6860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:54.492988 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:54.493281 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:54.494283 6860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:54.495604 6860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:54.495677 6860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:54.495853 6860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:54.495916 6860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:54.530160 6860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44693
I20260927 18:32:54.530216 6987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44693 every 8 connection(s)
I20260927 18:32:54.531437 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:54.539062 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6860
I20260927 18:32:54.539260 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:32:54.539683 389 external_mini_cluster.cc:1428] Setting key 6bc646bf4917be5e75846bbb39f65e39
Sep 27 18:32:54 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:54.551152 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:54.534153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33557 (local address 127.0.97.126:35643)
0927 18:32:54.534354 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:54.534359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:54.535559 (+ 1200us) server_negotiation.cc:407] Connection header received
0927 18:32:54.537262 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:54.537266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:54.537363 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:54.537476 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:54.538995 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.539884 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.541730 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.542313 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.542480 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:54.546011 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:54.546015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:54.546019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:54.546048 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:54.548458 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:54.549121 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:54.549125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:54.549127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:54.549239 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:54.549633 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:54.549640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:54.549646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:54.550014 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:54.550138 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:54.550742 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 18:32:54.550895 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:32:54.552071 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:54.534673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:33557)
0927 18:32:54.535380 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:54.535417 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:54.537003 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:54.537701 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:54.537710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:54.538277 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:54.538819 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:54.538837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.540029 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.540042 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:54.541598 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:54.541609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:54.542599 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:54.542609 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:54.542854 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:54.543547 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:54.543573 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:54.545748 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:54.548593 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:54.548600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:54.548612 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:54.549005 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:54.549333 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:54.549336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:54.549338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:54.549510 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:54.550118 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:54.550124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:54.550422 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 18:32:54.550651 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 18:32:54.551010 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":325,"threads_started":1}
I20260927 18:32:54.553610 6988 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:54.553974 6988 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:54.554821 6988 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
I20260927 18:32:54.556137 6532 ts_manager.cc:194] Registered new tserver with Master: 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
I20260927 18:32:54.556849 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:33557
I20260927 18:32:54.557260 6721 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
I20260927 18:32:54.567299 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.125.keytab.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.125.keytab.
I20260927 18:32:54.613488 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:41753
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35643,127.0.97.125:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:54.933508 6993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:54.933861 6993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:54.934002 6993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:54.945132 6993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:54.945235 6993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:54.945278 6993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:54.945333 6993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:54.945377 6993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:54.955163 6993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35643,127.0.97.125:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
W20260927 18:32:54.955448 6993 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.126:35643,127.0.97.125:41753'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:32:54.956408 6993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:54.957643 6993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:54.969141 6999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:54.969182 6998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:54.969671 7001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:54.970031 6993 server_base.cc:1034] running on GCE node
I20260927 18:32:54.970824 6993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:54.972230 6993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:54.973443 6993 hybrid_clock.cc:648] HybridClock initialized: now 1790533974973370 us; error 51 us; skew 500 ppm
I20260927 18:32:54.973810 6993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:54 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:54.977142 6993 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260927 18:32:54.978698 6993 webserver.cc:460] Webserver started at http://127.0.97.125:44227/ using document root <none> and password file <none>
I20260927 18:32:54.979359 6993 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:54.979487 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:54.979784 6993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:54.982800 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/instance:
uuid: "92523705747747c5b96d24aee4daccde"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "662c4f0646cd57829f3003da63f69c86"
server_key_iv: "1a3f6e671ec9f3d1236556cb019c7a2e"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.983556 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal/instance:
uuid: "92523705747747c5b96d24aee4daccde"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "662c4f0646cd57829f3003da63f69c86"
server_key_iv: "1a3f6e671ec9f3d1236556cb019c7a2e"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.988365 6993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 18:32:54.991753 7008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:54.993042 6993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:54.993227 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "92523705747747c5b96d24aee4daccde"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "662c4f0646cd57829f3003da63f69c86"
server_key_iv: "1a3f6e671ec9f3d1236556cb019c7a2e"
server_key_version: "encryptionkey@0"
I20260927 18:32:54.993439 6993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:55.006966 6993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:55.011273 6993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:55.011550 6993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:55.030122 6993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:41753
I20260927 18:32:55.030143 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:41753 every 8 connection(s)
I20260927 18:32:55.031461 6993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
I20260927 18:32:55.032769 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 6993
I20260927 18:32:55.035877 7060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:55.042506 7060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:55.055604 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.035062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.97.125:41753)
0927 18:32:55.035773 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.035786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.035822 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:32:55.037375 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.037395 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.037744 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.038197 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.039616 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.040912 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.042225 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.042915 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.043129 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.047272 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.047286 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.047297 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.047343 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.050872 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.051643 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.051650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.051657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.051847 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.052206 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.052211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.052215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.052843 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.053102 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.054184 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.054582 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":340,"threads_started":1}
I20260927 18:32:55.057354 6855 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
I20260927 18:32:55.060811 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.044829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45284)
0927 18:32:55.045698 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.045742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.045953 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.046333 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.046339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.046957 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.047303 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.047314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.048268 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.048274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.050124 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.050138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.050866 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.050872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.051066 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.051653 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.051739 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.054254 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.057443 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.057454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.057461 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.058094 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.058798 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.058805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.058809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.058966 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.059639 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.059646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.059829 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.060044 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.060508 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"spinlock_wait_cycles":15744,"thread_start_us":534,"threads_started":1}
I20260927 18:32:55.060817 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.044898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.0.97.126:35643)
0927 18:32:55.045151 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.045160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.045845 (+ 685us) server_negotiation.cc:407] Connection header received
0927 18:32:55.046088 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.046099 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.046161 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.046278 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.047447 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.048142 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.050256 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.050675 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.050776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.054498 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.054503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.054506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.054545 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.057307 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.058500 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.058505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.058508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.058663 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.059103 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.059108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.059118 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.059514 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.059669 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.060248 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.060511 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:32:55.062361 7060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260927 18:32:55.079914 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.063101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59600)
0927 18:32:55.063300 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.063319 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.063443 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.063738 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.063742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.064089 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.064204 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.064211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.065127 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.065135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.066059 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.066070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.066713 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.066718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.066968 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.072704 (+ 5736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.072721 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.074774 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.077548 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.077555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.077560 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.077971 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.078342 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.078347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.078350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.078465 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.078947 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.078951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.079076 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.079204 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.079492 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260927 18:32:55.079974 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.063147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.0.97.125:41753)
0927 18:32:55.063300 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.063306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.063389 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:32:55.063573 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.063580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.063632 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.063725 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.064327 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.065013 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.066194 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.066580 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.066705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.074952 (+ 8247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.074958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.074967 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.074994 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.077417 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.078086 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.078094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.078097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.078226 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.078570 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.078575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.078578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.078810 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.078954 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.079290 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.079637 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":45}
I20260927 18:32:55.084863 7060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Bootstrap starting.
I20260927 18:32:55.088002 7060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:55.089063 7060 log.cc:824] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:55.091738 7060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: No bootstrap required, opened a new log
I20260927 18:32:55.098722 7060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } }
I20260927 18:32:55.099041 7060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:55.099143 7060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92523705747747c5b96d24aee4daccde, State: Initialized, Role: FOLLOWER
I20260927 18:32:55.099695 7060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } }
I20260927 18:32:55.100464 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } }
I20260927 18:32:55.100733 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:55.101418 7060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:32:55.109747 7077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:32:55.109899 7077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:32:55.112548 6993 master_runner.cc:428] Detected that this master 92523705747747c5b96d24aee4daccde is joining an existing cluster
I20260927 18:32:55.112654 6993 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.125:41753
WARNING: no policy specified for oryx/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260927 18:32:55.138011 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.120236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59606)
0927 18:32:55.121190 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.121207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.121309 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.121741 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.121745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.122552 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.123267 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.123286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.124787 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.124793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.126669 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.126683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.127876 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.127883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.128108 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.128853 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.128875 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.130004 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.134907 (+ 4903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.134911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.134916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.135439 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.135965 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.135970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.135973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.136134 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.136820 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.136824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.136981 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.137133 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.137468 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":65,"thread_start_us":668,"threads_started":1}
I20260927 18:32:55.138118 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.119969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.97.125:41753)
0927 18:32:55.120312 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.120319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.121294 (+ 975us) server_negotiation.cc:407] Connection header received
0927 18:32:55.121416 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.121422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.121494 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.121601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.123425 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.124576 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.126810 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.127681 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.127818 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.130522 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.130528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.130533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.130570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.134768 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.135556 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.135561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.135565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.135697 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.136246 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.136251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.136254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.136595 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.136759 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.137269 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.137612 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":187}
I20260927 18:32:55.138659 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.119791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.97.126:35643)
0927 18:32:55.119978 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.119987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.121015 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 18:32:55.121298 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.121304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.121386 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.121458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.123428 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.124659 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.128275 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.129099 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.129211 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.131964 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.131973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.131992 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.132040 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.135661 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.136400 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.136406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.136409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.136552 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.136951 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.136955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.136959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.137311 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.137464 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.138059 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.138280 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.124@KRBTEST.COM" created.
I20260927 18:32:55.140199 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.119843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45298)
0927 18:32:55.120880 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.120904 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.121138 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.121677 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.121684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.122540 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.123251 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.123266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.124833 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.124844 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.128120 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.128133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.129522 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.129532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.129767 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.130481 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.130508 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.131732 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.135832 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.135840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.135845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.136263 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.136664 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.136669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.136672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.136839 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.137532 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.137540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.137743 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.139306 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.139668 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":138,"thread_start_us":667,"threads_started":1}
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260927 18:32:55.165196 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.147175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.0.97.125:41753)
0927 18:32:55.148197 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.148205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.148226 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:55.148307 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.148315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.148429 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.148553 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.149882 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.151099 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.152977 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.153828 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.153961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.158280 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.158289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.158295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.158342 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.162160 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.163017 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.163025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.163055 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.163199 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.163627 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.163633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.163638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.163956 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.164132 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.164576 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.164799 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884}
I20260927 18:32:55.165248 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.147194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59620)
0927 18:32:55.147838 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.147888 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.148061 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.148601 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.148611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.149426 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.149700 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.149712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.151237 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.151243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.152825 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.152839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.154014 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.154021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.154233 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.155055 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.155080 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.158069 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.162313 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.162323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.162329 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.162872 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.163322 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.163328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.163331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.163512 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.164108 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.164116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.164292 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.164473 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.164832 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":250,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.124.keytab.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.124.keytab.
I20260927 18:32:55.167148 6532 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.125:41753: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "92523705747747c5b96d24aee4daccde"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.125"
port: 41753
}
attrs {
promote: true
}
}
dest_uuid: "c310d59190374de6940119e43dcbc32f"
cas_config_opid_index: -1
I20260927 18:32:55.168116 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:43051
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:32:55.168450 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } }
I20260927 18:32:55.178267 7035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:55.182231 7089 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 92523705747747c5b96d24aee4daccde (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.183979 7035 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:32:55.190456 7090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:55.215344 7089 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.125:41753
I20260927 18:32:55.216123 6993 master.cc:537] Master@127.0.97.125:41753 shutting down...
I20260927 18:32:55.217350 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.217669 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.229957 7035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 92523705747747c5b96d24aee4daccde (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.249673 7101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.250038 7101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:55.250185 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: New leader c310d59190374de6940119e43dcbc32f. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.250422 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:55.250484 7097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.250825 7097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:55.253540 6993 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Raft consensus shutting down.
I20260927 18:32:55.254030 6993 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Raft consensus is shut down!
I20260927 18:32:55.254189 6993 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: stopping tablet replica
I20260927 18:32:55.271605 6993 master.cc:552] Master@127.0.97.125:41753 shutdown complete.
I20260927 18:32:55.271704 6993 master_runner.cc:305] Clearing existing system tablet
I20260927 18:32:55.272116 6993 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:55.272219 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:55.275239 6993 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 18:32:55.281435 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:55.281908 6993 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:55.282203 6993 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:32:55.282291 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "92523705747747c5b96d24aee4daccde"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "662c4f0646cd57829f3003da63f69c86"
server_key_iv: "1a3f6e671ec9f3d1236556cb019c7a2e"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.286767 6993 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:55.293649 6993 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:32:55.293853 6993 log.cc:1196] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:32:55.294376 6993 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Deleting consensus metadata
I20260927 18:32:55.294652 6993 master_runner.cc:315] Copying system tablet from 127.0.97.126:35643
I20260927 18:32:55.297410 6993 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:35643
I20260927 18:32:55.313124 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.298223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.0.97.126:35643)
0927 18:32:55.298397 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.298402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.298929 (+ 527us) server_negotiation.cc:407] Connection header received
0927 18:32:55.299166 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.299173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.299267 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.299378 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.300381 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.301517 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.303363 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.304051 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.304180 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.306436 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.306443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.306448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.306486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.310083 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.310989 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.310995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.310999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.311153 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.311581 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.311586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.311590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.311928 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.312082 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.312516 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.312764 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:55.313174 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.298192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45310)
0927 18:32:55.298807 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.298830 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.298981 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.299421 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.299426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.299923 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.300220 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.300231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.301666 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.301671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.303200 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.303218 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.304196 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.304202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.304388 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.305139 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.305162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.306248 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.310224 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.310231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.310238 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.310859 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.311273 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.311278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.311281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.311458 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.312053 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.312058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.312241 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.312402 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.312732 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":335,"threads_started":1}
I20260927 18:32:55.314462 6552 tablet_copy_service.cc:140] P c310d59190374de6940119e43dcbc32f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 92523705747747c5b96d24aee4daccde ({username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:45310)
I20260927 18:32:55.314714 6552 tablet_copy_service.cc:161] P c310d59190374de6940119e43dcbc32f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 92523705747747c5b96d24aee4daccde at {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:45310: session id = 92523705747747c5b96d24aee4daccde-00000000000000000000000000000000
I20260927 18:32:55.319152 6552 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:32:55.321457 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:32:55.328478 6993 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: tablet copy: Starting download of 0 data blocks...
I20260927 18:32:55.328824 6993 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: tablet copy: Starting download of 1 WAL segments...
I20260927 18:32:55.331553 6993 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:32:55.335671 6552 tablet_copy_service.cc:342] P c310d59190374de6940119e43dcbc32f: Request end of tablet copy session 92523705747747c5b96d24aee4daccde-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:45310
I20260927 18:32:55.335872 6552 tablet_copy_service.cc:433] P c310d59190374de6940119e43dcbc32f: ending tablet copy session 92523705747747c5b96d24aee4daccde-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 92523705747747c5b96d24aee4daccde
I20260927 18:32:55.347179 6993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:55.349756 7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:55.362246 7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:55.367786 6993 server_base.cc:1034] running on GCE node
W20260927 18:32:55.368485 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:55.369077 6993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:55.369686 6993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:55.370898 6993 hybrid_clock.cc:648] HybridClock initialized: now 1790533975370838 us; error 40 us; skew 500 ppm
I20260927 18:32:55.371120 6993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:55.373387 6993 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260927 18:32:55.374652 6993 webserver.cc:460] Webserver started at http://127.0.97.125:33095/ using document root <none> and password file <none>
I20260927 18:32:55.375061 6993 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:55.375156 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:55.378552 6993 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 18:32:55.380627 7120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:55.381064 6993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 18:32:55.381210 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "92523705747747c5b96d24aee4daccde"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "662c4f0646cd57829f3003da63f69c86"
server_key_iv: "1a3f6e671ec9f3d1236556cb019c7a2e"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.381412 6993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:55.404606 6993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:55.408967 6993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:55.440542 6993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:41753
I20260927 18:32:55.440572 7171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:41753 every 8 connection(s)
I20260927 18:32:55.441545 6993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
I20260927 18:32:55.447532 7172 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:32:55.451885 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Bootstrap starting.
I20260927 18:32:55.466610 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:32:55.467213 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Bootstrap complete.
I20260927 18:32:55.468654 7172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } }
I20260927 18:32:55.468914 7172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92523705747747c5b96d24aee4daccde, State: Initialized, Role: LEARNER
I20260927 18:32:55.469190 7172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } }
I20260927 18:32:55.470355 7172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:55.470083 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.470639 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:55.478838 7186 catalog_manager.cc:1259] Loaded cluster ID: b7516ade86d64b1f93f405d4f89955e4
I20260927 18:32:55.478983 7186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: loading cluster ID for follower catalog manager: success
I20260927 18:32:55.482888 7186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: acquiring CA information for follower catalog manager: success
I20260927 18:32:55.485977 7186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:32:55.559744 6988 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
W20260927 18:32:55.589447 7091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:55.589728 7091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:55.589901 7091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:55.597532 7091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:32:55.597632 7091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:55.597728 7091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:55.597828 7091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:32:55.597883 7091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:32:55.608289 7091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:55.609606 7091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:55.610941 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:55.622560 7191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:55.622568 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:55.622689 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:55.623433 7091 server_base.cc:1034] running on GCE node
I20260927 18:32:55.624105 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:55.625510 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:55.626695 7091 hybrid_clock.cc:648] HybridClock initialized: now 1790533975626633 us; error 63 us; skew 500 ppm
I20260927 18:32:55.626977 7091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:55.630460 7091 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260927 18:32:55.631642 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.631937 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.631981 7091 webserver.cc:460] Webserver started at http://127.0.97.124:37275/ using document root <none> and password file <none>
I20260927 18:32:55.632772 7091 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:55.632895 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:55.633245 7091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:32:55.636991 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/instance:
uuid: "3f1c896d2bf84814b4113c102af6ac1e"
format_stamp: "Formatted at 2026-09-27 18:32:55 on dist-test-slave-n3p4"
server_key: "3c29ffbb1143a298cdf596ec61738783"
server_key_iv: "ea28f042c67abf83979835d1f3ceabc7"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.637771 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal/instance:
uuid: "3f1c896d2bf84814b4113c102af6ac1e"
format_stamp: "Formatted at 2026-09-27 18:32:55 on dist-test-slave-n3p4"
server_key: "3c29ffbb1143a298cdf596ec61738783"
server_key_iv: "ea28f042c67abf83979835d1f3ceabc7"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.639091 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.632176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.0.97.125:41753)
0927 18:32:55.632730 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.632734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.632754 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:32:55.632812 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.632816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.632926 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.633067 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:55.633514 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.634514 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.636806 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.637733 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.637901 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.638146 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.638731 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.638830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":308,"threads_started":1}
I20260927 18:32:55.639112 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.631962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59626)
0927 18:32:55.632240 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.632260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.632383 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.633128 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.633133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.633163 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:55.633367 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.633380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.634662 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.634669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.636654 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.636671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.638308 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.638314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.638539 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.638602 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.638701 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.638845 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 18:32:55.640065 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: New leader c310d59190374de6940119e43dcbc32f. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.640306 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:55.641033 7089 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: attempting to promote NON_VOTER 92523705747747c5b96d24aee4daccde to VOTER
I20260927 18:32:55.641757 7089 consensus_queue.cc:237] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } }
I20260927 18:32:55.643440 7147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 LEARNER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:32:55.643976 7092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:32:55.644693 7091 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 18:32:55.646860 7089 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92523705747747c5b96d24aee4daccde (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } }
I20260927 18:32:55.647583 7147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92523705747747c5b96d24aee4daccde (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } }
I20260927 18:32:55.648411 7203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:55.649847 7091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:55.650014 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "3f1c896d2bf84814b4113c102af6ac1e"
format_stamp: "Formatted at 2026-09-27 18:32:55 on dist-test-slave-n3p4"
server_key: "3c29ffbb1143a298cdf596ec61738783"
server_key_iv: "ea28f042c67abf83979835d1f3ceabc7"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.650204 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:55.650550 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } }
I20260927 18:32:55.650763 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.651036 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } }
I20260927 18:32:55.651242 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.651362 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } }
I20260927 18:32:55.651595 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:55.673602 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:55.678282 7091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:55.678586 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:55.699507 7254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:43051 every 8 connection(s)
I20260927 18:32:55.699508 7091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:43051
I20260927 18:32:55.700932 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
I20260927 18:32:55.704610 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:55.706001 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 7091
I20260927 18:32:55.713078 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:32:55.731377 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.708221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.97.124:43051)
0927 18:32:55.708908 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.708920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.708963 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:32:55.710730 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.710751 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.711128 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.711542 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.712949 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.714873 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.716557 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.717596 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.717827 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.722994 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.723009 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.723028 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.723077 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.726763 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.727581 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.727592 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.727601 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.727793 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.728172 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.728176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.728178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.728917 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.729208 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.729590 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.730058 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260927 18:32:55.732487 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.716021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45314)
0927 18:32:55.716772 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.716810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.716977 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.717324 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.717330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.718186 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.718412 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.718420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.719492 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.719499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.721250 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.721261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.722358 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.722366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.722570 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.723266 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.723295 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.726215 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.729933 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.729941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.729946 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.730358 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.730765 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.730772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.730775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.730907 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.731560 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.731571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.731754 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.731958 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.732196 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":371,"threads_started":1}
I20260927 18:32:55.732587 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.716114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.0.97.126:35643)
0927 18:32:55.716293 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.716301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.716936 (+ 635us) server_negotiation.cc:407] Connection header received
0927 18:32:55.717101 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.717105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.717168 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.717245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.718577 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.719328 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.721401 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.722111 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.722213 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.726412 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.726419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.726434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.726492 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.729769 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.730487 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.730494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.730498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.730649 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.731018 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.731023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.731025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.731356 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.731517 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.732078 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.732289 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:32:55.734000 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260927 18:32:55.746920 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.734769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.0.97.125:41753)
0927 18:32:55.734946 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.734953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.734976 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:55.735153 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.735158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.735230 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.735334 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.736099 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.737298 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.738641 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.739068 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.739207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.742051 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.742057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.742063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.742098 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.744411 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.745047 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.745052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.745056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.745191 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.745537 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.745544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.745547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.745866 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.746010 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.746366 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.746612 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:32:55.746915 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.734730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59636)
0927 18:32:55.734908 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.734935 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.735037 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.735351 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.735355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.735811 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.735939 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.735967 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.737433 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.737441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.738496 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.738507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.739223 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.739228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.739412 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.739919 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.739953 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.741873 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.744546 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.744553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.744557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.744923 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.745296 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.745302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.745306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.745437 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.745976 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.745981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.746125 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.746283 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.746563 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:32:55.748687 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260927 18:32:55.761266 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.749467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.0.1:41864)
0927 18:32:55.749700 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.749716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.749868 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.750190 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.750193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.750540 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.750675 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.750683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.751581 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.751584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.752374 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.752383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.753094 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.753099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.753282 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.753944 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.753962 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.756099 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.758865 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.758869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.758872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.759275 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.759686 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.759690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.759692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.759848 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.760444 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.760448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.760651 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.760773 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.760946 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 18:32:55.761348 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.749619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.0.97.124:43051)
0927 18:32:55.749819 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.749825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.749842 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:32:55.749994 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.749998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.750074 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.750155 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.750789 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.751466 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.752535 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.752938 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.753034 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.756271 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.756275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.756278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.756305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.758756 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.759408 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.759413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.759416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.759546 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.759955 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.759958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.759961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.760336 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.760473 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.760889 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.761078 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:32:55.766693 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Bootstrap starting.
I20260927 18:32:55.770432 7255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:55.771730 7255 log.cc:824] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:55.774782 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: No bootstrap required, opened a new log
I20260927 18:32:55.781852 7255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } }
I20260927 18:32:55.782195 7255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:55.782318 7255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1c896d2bf84814b4113c102af6ac1e, State: Initialized, Role: FOLLOWER
I20260927 18:32:55.782932 7255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } }
I20260927 18:32:55.783771 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } } }
I20260927 18:32:55.784056 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:55.784823 7255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:32:55.794502 7272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:32:55.794646 7272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:32:55.799346 7091 master_runner.cc:428] Detected that this master 3f1c896d2bf84814b4113c102af6ac1e is joining an existing cluster
I20260927 18:32:55.799484 7091 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.124:43051
I20260927 18:32:55.822798 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.804463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.0.97.125:41753)
0927 18:32:55.804680 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.804687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.806262 (+ 1575us) server_negotiation.cc:407] Connection header received
0927 18:32:55.806453 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.806458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.806538 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.806640 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.808694 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.809826 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.812039 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.812655 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.812791 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.815383 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.815390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.815395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.815438 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.819290 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.820047 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.820052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.820056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.820197 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.820667 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.820674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.820677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.821053 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.821337 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.822173 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.822398 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:32:55.823953 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.803841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59652)
0927 18:32:55.806150 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.806183 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.806325 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.807268 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.807272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.808072 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.808523 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.808540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.809998 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.810004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.811863 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.811878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.813046 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.813052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.813212 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.813999 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.814022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.815193 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.819387 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.819392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.819396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.819915 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.820319 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.820323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.820327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.820530 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.821401 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.821406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.821667 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.821838 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.823191 (+ 1353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"mutex_wait_us":110,"thread_start_us":1159,"threads_started":1}
I20260927 18:32:55.827184 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.803974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.0.1:41876)
0927 18:32:55.806728 (+ 2754us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.806745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.806858 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.807585 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.807590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.808196 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.808553 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.808560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.810020 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.810028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.811893 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.811903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.816114 (+ 4211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.816121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.816375 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.818051 (+ 1676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.818078 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.819170 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.823748 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.823753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.823757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.824718 (+ 961us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.825148 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.825153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.825158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.825319 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.826090 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.826095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.826261 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.826396 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.826693 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2644,"mutex_wait_us":132,"thread_start_us":2459,"threads_started":1}
I20260927 18:32:55.827291 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.803884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.0.97.124:43051)
0927 18:32:55.804088 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.804098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.807148 (+ 3050us) server_negotiation.cc:407] Connection header received
0927 18:32:55.807208 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.807216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.807283 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.807386 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.808673 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.809825 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.814954 (+ 5129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.815722 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.815818 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.819364 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.819370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.819375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.819414 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.823616 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.824837 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.824842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.824845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.825022 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.825427 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.825433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.825438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.825879 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.826026 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.826556 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.826852 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":51}
I20260927 18:32:55.834460 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.804859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45324)
0927 18:32:55.819671 (+ 14812us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.819688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.819898 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.820457 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.820462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.821231 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.821605 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.821621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.823108 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.823115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.824872 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.824888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.826508 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.826515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.826695 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.827317 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.827342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.828421 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.832028 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.832031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.832034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.832418 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.832862 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.832865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.832867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.833017 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.833600 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.833603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.833759 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.833954 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.834147 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14659,"thread_start_us":12900,"threads_started":1}
I20260927 18:32:55.834621 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.803737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.0.97.126:35643)
0927 18:32:55.804123 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.804130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.819789 (+ 15659us) server_negotiation.cc:407] Connection header received
0927 18:32:55.820066 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.820072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.820174 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.820251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.821806 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.822894 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.825059 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.825672 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.825773 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.828610 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.828619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.828624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.828663 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.831884 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.832538 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.832546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.832551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.832716 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.833125 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.833130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.833134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.833450 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.833584 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.834066 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.834279 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260927 18:32:55.837616 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6593
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Sep 27 18:32:55 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260927 18:32:55.855446 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.840068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.0.1:41878)
0927 18:32:55.840312 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.840343 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.840486 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.840833 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.840841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.841197 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.841360 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.841368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.842481 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.842486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.843417 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.843428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.844454 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.844461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.844634 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.845232 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.845257 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.848466 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.852508 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.852516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.852520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.853005 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.853421 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.853426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.853429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.853571 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.854278 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.854283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.854443 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.854609 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.854905 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 18:32:55.855808 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.840115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.0.97.124:43051)
0927 18:32:55.840256 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.840261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.840418 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:32:55.840636 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.840639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.840698 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.840807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.841506 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.842352 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.843556 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.844286 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.844421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.848664 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.848670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.848675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.848711 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.852376 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.853136 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.853143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.853149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.853283 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.853702 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.853707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.853710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.854098 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.854246 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.854720 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.854936 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:32:55.857172 6532 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.124:43051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.124"
port: 43051
}
attrs {
promote: true
}
}
dest_uuid: "c310d59190374de6940119e43dcbc32f"
cas_config_opid_index: 6
I20260927 18:32:55.857977 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } }
I20260927 18:32:55.858639 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.838954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.0.97.65:43871)
0927 18:32:55.839579 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.839592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.839622 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:32:55.839694 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.839703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.839997 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.841855 (+ 1858us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.842114 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.843688 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.845403 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.846073 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.846227 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.851445 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.851455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.851469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.851503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.855208 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.856202 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.856211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.856214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.856357 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.856807 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.856812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.856816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.857269 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.857546 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.857996 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.858246 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":299,"threads_started":1}
I20260927 18:32:55.860210 6716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:55.860520 7147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:32:55.860777 7230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:55.861204 7092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 18:32:55.863729 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.864409 7230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:32:55.865912 7093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:55.868009 7147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.874286 7092 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.124:43051
I20260927 18:32:55.874910 7091 master.cc:537] Master@127.0.97.124:43051 shutting down...
I20260927 18:32:55.874874 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.875180 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.875531 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.875768 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:55.876959 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.877262 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:55.887476 7230 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } }
W20260927 18:32:55.887778 7230 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } }
I20260927 18:32:55.913874 7230 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 92523705747747c5b96d24aee4daccde (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.918334 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.918735 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:32:55.918776 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c310d59190374de6940119e43dcbc32f. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.918996 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:32:55.919137 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: true } } }
I20260927 18:32:55.919467 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:32:55.920650 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:55.920926 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
W20260927 18:32:55.921890 6519 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 18:32:55.922962 7091 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 18:32:55.923386 7091 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 18:32:55.923571 7091 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: stopping tablet replica
I20260927 18:32:55.931740 7091 master.cc:552] Master@127.0.97.124:43051 shutdown complete.
I20260927 18:32:55.931816 7091 master_runner.cc:305] Clearing existing system tablet
I20260927 18:32:55.932263 7091 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:55.932360 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260927 18:32:55.948575 6519 consensus_peers.cc:487] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f -> Peer 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124:43051): Couldn't send request to peer 3f1c896d2bf84814b4113c102af6ac1e. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:32:55.954447 7091 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.010s sys 0.011s
I20260927 18:32:55.959865 7291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:55.960348 7091 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:55.960595 7091 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:32:55.960687 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "3f1c896d2bf84814b4113c102af6ac1e"
format_stamp: "Formatted at 2026-09-27 18:32:55 on dist-test-slave-n3p4"
server_key: "3c29ffbb1143a298cdf596ec61738783"
server_key_iv: "ea28f042c67abf83979835d1f3ceabc7"
server_key_version: "encryptionkey@0"
I20260927 18:32:55.965986 7091 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:55.972857 7091 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:32:55.973001 7091 log.cc:1196] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:32:55.973580 7091 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Deleting consensus metadata
I20260927 18:32:55.974000 7091 master_runner.cc:315] Copying system tablet from 127.0.97.126:35643
I20260927 18:32:55.983582 7091 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:35643
I20260927 18:32:55.998742 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.984395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.0.97.126:35643)
0927 18:32:55.984660 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:55.984666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:55.984989 (+ 323us) server_negotiation.cc:407] Connection header received
0927 18:32:55.985161 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:55.985167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:55.985249 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:55.985372 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:55.986313 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.987534 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.989488 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.990222 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.990347 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.992541 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:55.992548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:55.992553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:55.992592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:55.995962 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.996671 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.996677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.996681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.996826 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:55.997218 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:55.997225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:55.997229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:55.997561 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:55.997721 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:55.998136 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 18:32:55.998351 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:32:55.998781 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:55.984358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45340)
0927 18:32:55.984862 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:55.984884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:55.985039 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:55.985394 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:55.985399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:55.985899 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:55.986152 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.986176 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.987689 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.987694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.989354 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:55.989367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:55.990368 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:55.990374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:55.990548 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:55.991286 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:55.991311 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:55.992331 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:55.996092 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.996098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.996102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.996559 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.996941 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:55.996946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:55.996950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:55.997113 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:55.997675 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:55.997680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:55.997873 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 18:32:55.998029 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 18:32:55.998317 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260927 18:32:55.999830 6552 tablet_copy_service.cc:140] P c310d59190374de6940119e43dcbc32f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f1c896d2bf84814b4113c102af6ac1e ({username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:45340)
I20260927 18:32:55.999998 6552 tablet_copy_service.cc:161] P c310d59190374de6940119e43dcbc32f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f1c896d2bf84814b4113c102af6ac1e at {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:45340: session id = 3f1c896d2bf84814b4113c102af6ac1e-00000000000000000000000000000000
I20260927 18:32:56.004055 6552 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:32:56.005906 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:32:56.012583 7091 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: tablet copy: Starting download of 0 data blocks...
I20260927 18:32:56.012974 7091 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: tablet copy: Starting download of 1 WAL segments...
I20260927 18:32:56.014670 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6593
I20260927 18:32:56.015272 7091 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:32:56.019590 6552 tablet_copy_service.cc:342] P c310d59190374de6940119e43dcbc32f: Request end of tablet copy session 3f1c896d2bf84814b4113c102af6ac1e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:45340
I20260927 18:32:56.019737 6552 tablet_copy_service.cc:433] P c310d59190374de6940119e43dcbc32f: ending tablet copy session 3f1c896d2bf84814b4113c102af6ac1e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f1c896d2bf84814b4113c102af6ac1e
I20260927 18:32:56.031852 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:56.034663 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:56.038021 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:43871
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35643
--webserver_port=34341
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:56.044410 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:56.048230 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:56.049715 7091 server_base.cc:1034] running on GCE node
I20260927 18:32:56.050069 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:56.050740 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:56.051999 7091 hybrid_clock.cc:648] HybridClock initialized: now 1790533976051961 us; error 35 us; skew 500 ppm
I20260927 18:32:56.052233 7091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:56.054262 7091 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260927 18:32:56.059007 7091 webserver.cc:460] Webserver started at http://127.0.97.124:36433/ using document root <none> and password file <none>
I20260927 18:32:56.059412 7091 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:56.059513 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:56.062400 7091 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:32:56.064002 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:56.064379 7091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:32:56.064514 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "3f1c896d2bf84814b4113c102af6ac1e"
format_stamp: "Formatted at 2026-09-27 18:32:55 on dist-test-slave-n3p4"
server_key: "3c29ffbb1143a298cdf596ec61738783"
server_key_iv: "ea28f042c67abf83979835d1f3ceabc7"
server_key_version: "encryptionkey@0"
I20260927 18:32:56.064702 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:56.080796 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:56.084501 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:56.107872 7091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:43051
I20260927 18:32:56.107899 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:43051 every 8 connection(s)
I20260927 18:32:56.108800 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
I20260927 18:32:56.114605 7361 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:32:56.116989 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Bootstrap starting.
I20260927 18:32:56.128356 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:32:56.128796 7361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: Bootstrap complete.
I20260927 18:32:56.129748 7361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } }
I20260927 18:32:56.130048 7361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f1c896d2bf84814b4113c102af6ac1e, State: Initialized, Role: LEARNER
I20260927 18:32:56.130287 7361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } }
I20260927 18:32:56.134337 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:56.134653 7361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:32:56.134820 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:56.164243 7375 catalog_manager.cc:1259] Loaded cluster ID: b7516ade86d64b1f93f405d4f89955e4
I20260927 18:32:56.164376 7375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: loading cluster ID for follower catalog manager: success
I20260927 18:32:56.167078 7375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: acquiring CA information for follower catalog manager: success
I20260927 18:32:56.168740 7375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:32:56.443869 7303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:56.444145 7303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:56.444299 7303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:56.452232 7303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:56.452337 7303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:56.452531 7303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:32:56.463029 7303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:43871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=34341
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:56.464461 7303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:56.466060 7303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:56.480003 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:56.479995 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:56.479985 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:56.480850 7303 server_base.cc:1034] running on GCE node
I20260927 18:32:56.481600 7303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:56.482748 7303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:56.483958 7303 hybrid_clock.cc:648] HybridClock initialized: now 1790533976483896 us; error 46 us; skew 500 ppm
I20260927 18:32:56.484236 7303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:56.487797 7303 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260927 18:32:56.489676 7303 webserver.cc:460] Webserver started at http://127.0.97.65:34341/ using document root <none> and password file <none>
I20260927 18:32:56.490429 7303 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:56.490592 7303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:56.496250 7303 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 18:32:56.499760 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:56.501169 7303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:32:56.501360 7303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "7365fcbd04ca4282b63a4c0908b1b5fd"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "97fdb080a83a5a6a644199b3581fe286"
server_key_iv: "0a73769b098a16f46f04a2b8ed780b8d"
server_key_version: "encryptionkey@0"
I20260927 18:32:56.502463 7303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:56.504662 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:56.504941 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:56.523636 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.504841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.0.1:41888)
0927 18:32:56.516987 (+ 12146us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:56.517023 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:56.517224 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:56.518885 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:56.518890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:56.518918 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:56.519120 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.519130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.520355 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.520360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.521570 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.521582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.522787 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.522793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.522999 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.523040 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:32:56.523154 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:32:56.523298 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11956,"thread_start_us":12505,"threads_started":1}
I20260927 18:32:56.523636 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.504947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.0.97.124:43051)
0927 18:32:56.518542 (+ 13595us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:56.518556 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:56.518576 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:32:56.518640 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:56.518643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:56.518713 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:56.518855 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:56.519278 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.520207 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.521732 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.522616 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.522752 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.523010 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:56.523187 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 18:32:56.523315 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13439,"thread_start_us":13301,"threads_started":1}
I20260927 18:32:56.524550 7336 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 18:32:56.525375 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c310d59190374de6940119e43dcbc32f. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: true } } }
I20260927 18:32:56.525672 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: LEARNER
I20260927 18:32:56.526130 7392 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: attempting to promote NON_VOTER 3f1c896d2bf84814b4113c102af6ac1e to VOTER
I20260927 18:32:56.526765 7303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:56.526739 7392 consensus_queue.cc:237] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } }
I20260927 18:32:56.528501 7147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:32:56.528513 7336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 LEARNER]: Refusing update from remote peer c310d59190374de6940119e43dcbc32f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:32:56.529073 7392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:32:56.529349 7092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:32:56.532831 7303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:56.532584 7391 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.533141 7303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:56.533082 7147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.534606 7303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:56.536432 7303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:56.536532 7303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:56.536643 7303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:56.536710 7303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:56.537218 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.537582 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:56.537946 7336 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.538947 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.538066 7391 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.539597 7391 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:56.539485 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:56.542289 7403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.542793 7403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:56.542968 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c310d59190374de6940119e43dcbc32f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:56.543228 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:56.578496 7303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43871
I20260927 18:32:56.578537 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43871 every 8 connection(s)
I20260927 18:32:56.579885 7303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:32:56.584376 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 7303
I20260927 18:32:56.584687 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6727
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Sep 27 18:32:56 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
I20260927 18:32:56.623531 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.583395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57119 (local address 127.0.97.125:41753)
0927 18:32:56.600375 (+ 16980us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:56.600382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:56.600403 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:32:56.601708 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:56.601735 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:56.601877 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:56.601997 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:56.605920 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.607100 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.609282 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.610043 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610182 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.616246 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:56.616253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:56.616257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:56.616299 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:56.619937 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.620651 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.620657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.620661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.620778 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.621191 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.621198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.621203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.621469 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:56.621613 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:56.622890 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0927 18:32:56.623132 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16812,"spinlock_wait_cycles":5504,"thread_start_us":16773,"threads_started":1}
I20260927 18:32:56.624418 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.583210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58481 (local address 127.0.97.126:35643)
0927 18:32:56.600837 (+ 17627us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:56.600842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:56.600865 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:56.601677 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:56.601683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:56.601757 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:56.601875 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:56.606719 (+ 4844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.607903 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.609919 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.610664 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.617271 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:56.617278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:56.617282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:56.617322 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:56.620945 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.621768 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.621802 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.621807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.621940 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.622420 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.622425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.622816 (+ 391us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.623145 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:56.623259 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:56.623798 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 18:32:56.624022 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17497,"thread_start_us":17344,"threads_started":1}
I20260927 18:32:56.625159 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.584472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34839 (local address 127.0.97.124:43051)
0927 18:32:56.584734 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:56.584741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:56.594478 (+ 9737us) server_negotiation.cc:407] Connection header received
0927 18:32:56.601607 (+ 7129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:56.601615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:56.601704 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:56.601771 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:56.604817 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.606410 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.609277 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.609954 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610057 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.617223 (+ 7166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:56.617229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:56.617233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:56.617294 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:56.621619 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.622397 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.622406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.622410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.622566 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.623149 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.623154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.623158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.623479 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:56.623911 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:56.624412 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 18:32:56.624872 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 18:32:56.625571 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.583338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.97.65:57119)
0927 18:32:56.599588 (+ 16250us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:56.599614 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:56.601290 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:56.602301 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:56.602315 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:56.603479 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:56.604604 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.604631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.607269 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.607274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.609113 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.609127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.610479 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610490 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.610819 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.611809 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:56.611848 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:56.615971 (+ 4123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:56.620083 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.620093 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.620117 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.620533 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.620888 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.620892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.620896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.621083 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.621602 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:56.621611 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:56.622108 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 18:32:56.622448 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 18:32:56.624158 (+ 1710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16068,"mutex_wait_us":2100,"spinlock_wait_cycles":33408,"thread_start_us":9571,"threads_started":1}
I20260927 18:32:56.625697 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.585011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:58481)
0927 18:32:56.594904 (+ 9893us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:56.594921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:56.601259 (+ 6338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:56.602298 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:56.602311 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:56.603720 (+ 1409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:56.606564 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.606575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.608069 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.608076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.609705 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.609718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.611076 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.611082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.611249 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.611983 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:56.612014 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:56.616969 (+ 4955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:56.621105 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.621110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.621115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.621613 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.622093 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.622099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.622103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.622281 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.623350 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:56.623355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:56.623524 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 18:32:56.624668 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0927 18:32:56.624960 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9773,"mutex_wait_us":2398,"spinlock_wait_cycles":29440,"thread_start_us":8867,"threads_started":1}
I20260927 18:32:56.627031 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.583338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.97.65:34839)
0927 18:32:56.594303 (+ 10965us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:56.594363 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:56.601259 (+ 6896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:56.602298 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:56.602311 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:56.603477 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:56.604604 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.604630 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.606566 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.606573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.609113 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:56.609127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.610410 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:56.610822 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.611809 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:56.611850 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:56.616261 (+ 4411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:56.621735 (+ 5474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.621739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.621742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.622254 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.622843 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:56.622848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:56.622852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:56.623024 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:56.624005 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:56.624009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:56.624141 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 18:32:56.625607 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0927 18:32:56.626626 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10648,"mutex_wait_us":274,"spinlock_wait_cycles":12800,"thread_start_us":16263,"threads_started":1}
I20260927 18:32:56.627183 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.125:41753
I20260927 18:32:56.627301 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:56.627573 7511 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:56.627583 7512 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:56.628160 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.124:43051
I20260927 18:32:56.628281 7510 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:56.628715 7511 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
I20260927 18:32:56.628738 7510 heartbeater.cc:511] Master 127.0.97.124:43051 requested a full tablet report, sending...
I20260927 18:32:56.629595 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:56.585733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.0.97.66:39207)
0927 18:32:56.603589 (+ 17856us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:56.603598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:56.603631 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:32:56.603697 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:56.603707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:56.603967 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:56.604145 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:56.606779 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.608550 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.610543 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:56.612040 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:56.616002 (+ 3962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:56.617455 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:56.617470 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:56.617488 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:56.617539 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:56.624076 (+ 6537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.624965 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.624974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.624978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.625127 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:56.625574 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:56.625578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:56.625581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:56.626016 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:56.628975 (+ 2959us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:56.629088 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 18:32:56.629272 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17674,"thread_start_us":17489,"threads_started":1}
I20260927 18:32:56.630039 7512 heartbeater.cc:511] Master 127.0.97.125:41753 requested a full tablet report, sending...
I20260927 18:32:56.630985 6850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:56.631333 7326 ts_manager.cc:194] Registered new tserver with Master: 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871)
I20260927 18:32:56.632050 7137 ts_manager.cc:194] Registered new tserver with Master: 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871)
I20260927 18:32:56.632366 6532 ts_manager.cc:194] Re-registered known tserver with Master: 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871)
I20260927 18:32:56.633366 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:58481
I20260927 18:32:56.719154 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6727
I20260927 18:32:56.735877 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:39207
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35643
--webserver_port=34345
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:32:57.069651 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:57.070025 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:57.070197 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:57.077901 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:57.078047 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:57.078248 7522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:32:57.088249 7522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:39207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=34345
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:57.089668 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:57.091023 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:57.104236 7527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:57.104290 7528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:57.104324 7530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:57.105003 7522 server_base.cc:1034] running on GCE node
I20260927 18:32:57.105623 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:57.106593 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:57.107824 7522 hybrid_clock.cc:648] HybridClock initialized: now 1790533977107763 us; error 35 us; skew 500 ppm
I20260927 18:32:57.108183 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:57.112023 7522 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260927 18:32:57.113677 7522 webserver.cc:460] Webserver started at http://127.0.97.66:34345/ using document root <none> and password file <none>
I20260927 18:32:57.114485 7522 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:57.114615 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:57.121043 7522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:32:57.124159 7537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.125666 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:32:57.125873 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "dfae4e78bf9540788eb6ecd5dc311565"
format_stamp: "Formatted at 2026-09-27 18:32:53 on dist-test-slave-n3p4"
server_key: "ea4e55f20589dbb23a08a0321d6684e8"
server_key_iv: "f6d6cc3b2b2caf390e06ff280eae1134"
server_key_version: "encryptionkey@0"
I20260927 18:32:57.126969 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:57.145718 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:57.150113 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:57.150368 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:57.151317 7522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:57.152698 7522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:57.152791 7522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.152877 7522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:57.152940 7522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.188302 7522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39207
I20260927 18:32:57.188309 7649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39207 every 8 connection(s)
I20260927 18:32:57.190075 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:32:57.196476 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 7522
I20260927 18:32:57.196754 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6860
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260927 18:32:57.256312 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.193711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39831 (local address 127.0.97.125:41753)
0927 18:32:57.208869 (+ 15158us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.208876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.234718 (+ 25842us) server_negotiation.cc:407] Connection header received
0927 18:32:57.234802 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.234810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.234915 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.235033 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.238099 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239199 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.242086 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.242745 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.242849 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.248458 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.248465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.248470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.248531 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.252396 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.253452 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.253458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.253461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.253603 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.254175 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.254182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.254189 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.254548 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.254671 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.255700 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.255927 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15019,"thread_start_us":15171,"threads_started":1}
I20260927 18:32:57.256670 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.193970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58449 (local address 127.0.97.126:35643)
0927 18:32:57.236128 (+ 42158us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.236134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.236157 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:57.236219 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.236226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.236298 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.236891 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.238097 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239176 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.242084 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.242764 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.242876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.248971 (+ 6095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.248978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.248982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.249024 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.253155 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.253763 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.253769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.253772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.253970 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.254671 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.254675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.254679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.254945 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.255087 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.256100 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.256318 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42034,"thread_start_us":41939,"threads_started":1}
I20260927 18:32:57.258122 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.194010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:58449)
0927 18:32:57.208633 (+ 14623us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.208707 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.211145 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.236471 (+ 25326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.236475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.236927 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.237895 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.237923 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239357 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.239362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.241963 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.241980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.242997 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.243005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.243368 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.244405 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.244451 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.248739 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.253276 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.253281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.253285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.253625 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.254378 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.254382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.254385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.254537 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.255070 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.255075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.255250 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.255578 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.256139 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14173,"mutex_wait_us":284,"spinlock_wait_cycles":25600,"thread_start_us":41368,"threads_started":1}
I20260927 18:32:57.258157 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.194011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.97.66:39831)
0927 18:32:57.234379 (+ 40368us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.234405 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.234577 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.235655 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.235669 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.236716 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.237895 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.237923 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239357 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.239362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.241925 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.241940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.243125 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.243130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.243363 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.244146 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.244189 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.248223 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.252592 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.252605 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.252626 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.253309 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.253752 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.253758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.253762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.254036 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.254762 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.254770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.255232 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.256536 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.256817 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40163,"mutex_wait_us":342,"spinlock_wait_cycles":1280,"thread_start_us":15632,"threads_started":1}
I20260927 18:32:57.259106 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.193293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41801 (local address 127.0.97.124:43051)
0927 18:32:57.211308 (+ 18015us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.211316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.236016 (+ 24700us) server_negotiation.cc:407] Connection header received
0927 18:32:57.236292 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.236300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.236387 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.236485 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.238459 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239523 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.243080 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.243685 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.243785 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.249670 (+ 5885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.249679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.249684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.249728 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.253098 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.254373 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.254378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.254382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.254523 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.255481 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.255487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.255490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.255810 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.255949 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.257525 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.258581 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17871,"thread_start_us":17781,"threads_started":1}
I20260927 18:32:57.259248 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.198131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.97.67:44693)
0927 18:32:57.231988 (+ 33857us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.232002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.232044 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:32:57.232135 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.232147 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.232525 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.232698 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.237032 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.238689 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.240313 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.241110 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.241241 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.249714 (+ 8473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.249729 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.249749 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.249813 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.254369 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.255287 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.255298 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.255303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.255468 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.255899 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.255904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.255906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.256281 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.257308 (+ 1027us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.258157 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.258824 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33625,"thread_start_us":33455,"threads_started":1}
I20260927 18:32:57.259847 7652 heartbeater.cc:347] Connected to a master server at 127.0.97.125:41753
I20260927 18:32:57.260197 7652 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.260257 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.194687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.97.66:41801)
0927 18:32:57.235908 (+ 41221us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.235928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.236128 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.236705 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.236711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.238034 (+ 1323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.238283 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.238294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.239652 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.239657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.241955 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.241963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.244099 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.244105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.244296 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.244970 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.245003 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.249448 (+ 4445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.253774 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.253778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.253782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.254247 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.255134 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.255139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.255142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.255342 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.257090 (+ 1748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.257094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.257218 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.259344 (+ 2126us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.259604 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41065,"mutex_wait_us":162,"thread_start_us":39302,"threads_started":1}
I20260927 18:32:57.260838 7651 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:57.260962 7651 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.261317 7652 heartbeater.cc:511] Master 127.0.97.125:41753 requested a full tablet report, sending...
I20260927 18:32:57.261436 6983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:57.261514 7651 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
I20260927 18:32:57.262348 7650 heartbeater.cc:347] Connected to a master server at 127.0.97.124:43051
I20260927 18:32:57.262495 7650 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.262991 7650 heartbeater.cc:511] Master 127.0.97.124:43051 requested a full tablet report, sending...
I20260927 18:32:57.263198 7137 ts_manager.cc:194] Registered new tserver with Master: dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:57.264171 6532 ts_manager.cc:194] Re-registered known tserver with Master: dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:57.264240 7326 ts_manager.cc:194] Registered new tserver with Master: dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:57.265349 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:58449
I20260927 18:32:57.347877 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6860
I20260927 18:32:57.366365 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:44693
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35643
--webserver_port=36061
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:32:57.635926 7511 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
W20260927 18:32:57.706364 7663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:32:57.706627 7663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:32:57.706801 7663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:32:57.714300 7663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:32:57.714458 7663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:32:57.714629 7663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:32:57.724108 7663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:44693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=36061
--tserver_master_addrs=127.0.97.126:35643,127.0.97.125:41753,127.0.97.124:43051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:32:57.725507 7663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:32:57.726820 7663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:32:57.739910 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:57.739867 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:32:57.739952 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:32:57.741072 7663 server_base.cc:1034] running on GCE node
I20260927 18:32:57.741654 7663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:32:57.742621 7663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:32:57.743882 7663 hybrid_clock.cc:648] HybridClock initialized: now 1790533977743822 us; error 69 us; skew 500 ppm
I20260927 18:32:57.744275 7663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:32:57.747941 7663 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260927 18:32:57.749531 7663 webserver.cc:460] Webserver started at http://127.0.97.67:36061/ using document root <none> and password file <none>
I20260927 18:32:57.750313 7663 fs_manager.cc:359] Metadata directory not provided
I20260927 18:32:57.750451 7663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:32:57.756192 7663 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 18:32:57.758898 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.760300 7663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:32:57.760468 7663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "4d273b5452b34e25b8d51b42f0271c26"
format_stamp: "Formatted at 2026-09-27 18:32:54 on dist-test-slave-n3p4"
server_key: "41ec6c95633d94745fae419113dc7413"
server_key_iv: "9812db775b904cf4d4700c2fa73f4b93"
server_key_version: "encryptionkey@0"
I20260927 18:32:57.761672 7663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:32:57.783981 7663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:32:57.787770 7663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:32:57.787992 7663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:32:57.789009 7663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:32:57.790386 7663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:32:57.790473 7663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.790549 7663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:32:57.790601 7663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:32:57.823899 7663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44693
I20260927 18:32:57.823954 7790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44693 every 8 connection(s)
I20260927 18:32:57.825326 7663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:32:57.827409 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 7663
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260927 18:32:57.874989 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.830179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35379 (local address 127.0.97.124:43051)
0927 18:32:57.842906 (+ 12727us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.842912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.843538 (+ 626us) server_negotiation.cc:407] Connection header received
0927 18:32:57.845597 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.845605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.845688 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.845802 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.849644 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.851092 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.856515 (+ 5423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.857337 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.857456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.864398 (+ 6942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.864405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.864411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.864450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.871137 (+ 6687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.871947 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.871965 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.871970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.872099 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.872591 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.872596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.872602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.872936 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.873078 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.874381 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.874622 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12602,"thread_start_us":12874,"threads_started":1}
I20260927 18:32:57.876279 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.830628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.97.67:35379)
0927 18:32:57.843405 (+ 12777us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.843423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.845134 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.846250 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.846263 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.849088 (+ 2825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.849431 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.849445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.851318 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.851326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.854088 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.854107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.858794 (+ 4687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.858802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.859042 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.861566 (+ 2524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.861587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.864187 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.871296 (+ 7109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.871302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.871307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.871800 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.872219 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.872224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.872228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.872464 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.873157 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.873166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.873591 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.873949 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.874721 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12631,"thread_start_us":11364,"threads_started":1}
I20260927 18:32:57.876946 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.833521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52399 (local address 127.0.97.126:35643)
0927 18:32:57.834305 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.834314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.844125 (+ 9811us) server_negotiation.cc:407] Connection header received
0927 18:32:57.845462 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.845468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.845559 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.845697 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.848874 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.850447 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.856457 (+ 6010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.857306 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.857448 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.865152 (+ 7704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.865163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.865169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.865246 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.870021 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.871506 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.871514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.871521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.871688 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.873315 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.873323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.873329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.873714 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.873880 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.876344 (+ 2464us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.876569 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":520,"threads_started":1}
I20260927 18:32:57.877947 7791 heartbeater.cc:347] Connected to a master server at 127.0.97.124:43051
I20260927 18:32:57.878283 7791 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.878852 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.833521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:52399)
0927 18:32:57.844004 (+ 10483us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.844020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.845134 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.846250 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.846263 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.847622 (+ 1359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.848668 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.848699 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.854278 (+ 5579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.854286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.856296 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.856313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.857860 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.857870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.858239 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.859473 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.859512 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.864024 (+ 4512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.870219 (+ 6195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.870232 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.870313 (+ 81us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.871244 (+ 931us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.871832 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.871840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.871845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.873161 (+ 1316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.873945 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.873950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.874529 (+ 579us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.874726 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.875408 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10349,"thread_start_us":10198,"threads_started":1}
Sep 27 18:32:57 dist-test-slave-n3p4 krb5kdc[6487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260927 18:32:57.879258 7791 heartbeater.cc:511] Master 127.0.97.124:43051 requested a full tablet report, sending...
I20260927 18:32:57.879817 7792 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35643
I20260927 18:32:57.879946 7792 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.880420 7792 heartbeater.cc:511] Master 127.0.97.126:35643 requested a full tablet report, sending...
I20260927 18:32:57.881294 6532 ts_manager.cc:194] Re-registered known tserver with Master: 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
I20260927 18:32:57.881382 7326 ts_manager.cc:194] Registered new tserver with Master: 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
I20260927 18:32:57.882900 6532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:52399
I20260927 18:32:57.887422 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.828443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51829 (local address 127.0.97.125:41753)
0927 18:32:57.838144 (+ 9701us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.838148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.842565 (+ 4417us) server_negotiation.cc:407] Connection header received
0927 18:32:57.845674 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.845679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.845764 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.845862 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.865436 (+ 19574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.866730 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.872715 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.873346 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.873438 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.880343 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.880351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.880357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.880409 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.884308 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.885140 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.885145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.885149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.885290 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.885765 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.885770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.885774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.886157 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.886312 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.886825 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.887049 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9597,"thread_start_us":9492,"threads_started":1}
I20260927 18:32:57.888441 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.829137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.97.67:51829)
0927 18:32:57.842376 (+ 13239us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.842418 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.845526 (+ 3108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.846628 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.846633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.864954 (+ 18321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:32:57.865246 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.865261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.868325 (+ 3064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.868333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.872567 (+ 4234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.872579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.873806 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.873813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.874012 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.875853 (+ 1841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:32:57.875879 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:32:57.880090 (+ 4211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:32:57.884440 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.884447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.884450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.884995 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.885441 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:32:57.885446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:32:57.885450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:32:57.885630 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:32:57.886382 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:32:57.886386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:32:57.886528 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.887674 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.888002 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12964,"thread_start_us":13835,"threads_started":1}
I20260927 18:32:57.889367 7793 heartbeater.cc:347] Connected to a master server at 127.0.97.125:41753
I20260927 18:32:57.889529 7793 heartbeater.cc:464] Registering TS with master...
I20260927 18:32:57.889945 7793 heartbeater.cc:511] Master 127.0.97.125:41753 requested a full tablet report, sending...
I20260927 18:32:57.891114 7137 ts_manager.cc:194] Registered new tserver with Master: 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
I20260927 18:32:57.893496 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.847287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.0.97.126:35643)
0927 18:32:57.860224 (+ 12937us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.860230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.861439 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 18:32:57.861511 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.861517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.861594 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.861694 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.864416 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.866333 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.872009 (+ 5676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.874196 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.874345 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.884024 (+ 9679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.884033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.884039 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.884075 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.887723 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.890289 (+ 2566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.890296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.890300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.890434 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.891697 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.891703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.891706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.892043 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.892283 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.892877 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.893091 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12827,"thread_start_us":14046,"threads_started":1}
I20260927 18:32:57.897948 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.846968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.97.125:41753)
0927 18:32:57.858400 (+ 11432us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.858406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.875561 (+ 17155us) server_negotiation.cc:407] Connection header received
0927 18:32:57.877504 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.877514 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.877620 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.877738 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.882323 (+ 4585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.883691 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.886381 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.887050 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.887137 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.891151 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.891157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.891160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.891187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.894651 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.895506 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.895512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.895518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.895674 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.896181 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.896188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.896191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.896544 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.896781 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.897197 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.897546 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11308,"thread_start_us":11560,"threads_started":1}
I20260927 18:32:57.900590 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.854634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.97.124:43051)
0927 18:32:57.857158 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.857164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.881630 (+ 24466us) server_negotiation.cc:407] Connection header received
0927 18:32:57.881683 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.881689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.881767 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.881889 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:57.887859 (+ 5970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.888906 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.890895 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.891510 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.891601 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.894975 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:57.894982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:57.894986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:57.895023 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:57.898075 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.898687 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.898693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.898696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.898864 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:57.899266 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:57.899270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:57.899272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:57.899529 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:57.899726 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.900102 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.900271 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2373,"thread_start_us":2954,"threads_started":1}
I20260927 18:32:57.897677 6532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:57.905993 6532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:57.936285 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.924171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43871 (local address 127.0.0.1:43334)
0927 18:32:57.924706 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.924732 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.924864 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.925721 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.925725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.925747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:57.926245 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.926261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.928834 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.928842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.931880 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.931900 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.935180 (+ 3280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.935190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.935453 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.935504 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.935602 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.935791 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260927 18:32:57.937698 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.924390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.97.65:43871)
0927 18:32:57.925082 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.925092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.925120 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:32:57.925221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.925230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.925538 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.925745 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:57.926466 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.928630 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.932109 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.934585 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.934731 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.935020 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.937076 (+ 2056us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.937310 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":445,"threads_started":1}
I20260927 18:32:57.942308 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 (DEFAULT_TABLE table=test-table [id=67ad541b1944420e8d78256fa3fd146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:57.944118 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:57.952968 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.925466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39207 (local address 127.0.0.1:47904)
0927 18:32:57.936740 (+ 11274us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.936762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.936876 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.944641 (+ 7765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.944646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.944671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:57.944836 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.944844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.947377 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.947382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.949300 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.949313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.952159 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.952165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.952365 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.952400 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.952498 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.952644 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11121,"threads_started":1}
I20260927 18:32:57.953589 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.925739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.0.97.66:39207)
0927 18:32:57.943979 (+ 18240us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.943998 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.944028 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:32:57.944121 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.944128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.944445 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.944982 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:57.945170 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.946641 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.950240 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.952002 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.952125 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.952318 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.953156 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.953331 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18070,"thread_start_us":18002,"threads_started":1}
I20260927 18:32:57.958760 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.926372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44693 (local address 127.0.0.1:49940)
0927 18:32:57.947124 (+ 20752us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:57.947141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:57.947246 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:57.949593 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:57.949597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:57.949620 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:57.950019 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.950030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.952381 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.952386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.954847 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:57.954859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.957955 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.957962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:57.958155 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.958190 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:32:57.958271 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:32:57.958395 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20636,"thread_start_us":20481,"threads_started":1}
I20260927 18:32:57.958622 7585 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 (DEFAULT_TABLE table=test-table [id=67ad541b1944420e8d78256fa3fd146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:57.959555 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:57.926437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.0.97.67:44693)
0927 18:32:57.941635 (+ 15198us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:57.941641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:57.947254 (+ 5613us) server_negotiation.cc:407] Connection header received
0927 18:32:57.947364 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:57.947372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:57.947592 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:57.948100 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:57.950316 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.951919 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.955022 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:57.956895 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:57.957034 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:57.957232 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:57.959085 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0927 18:32:57.959260 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15070,"thread_start_us":15202,"threads_started":1}
I20260927 18:32:57.959868 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:57.961844 7828 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Bootstrap starting.
I20260927 18:32:57.963150 7726 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 (DEFAULT_TABLE table=test-table [id=67ad541b1944420e8d78256fa3fd146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:57.964502 7726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:57.965535 7828 tablet_bootstrap.cc:654] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:57.966671 7828 log.cc:824] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:57.969023 7828 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: No bootstrap required, opened a new log
I20260927 18:32:57.969292 7828 ts_tablet_manager.cc:1397] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 18:32:57.976408 7830 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Bootstrap starting.
I20260927 18:32:57.977738 7828 raft_consensus.cc:348] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:57.978578 7828 raft_consensus.cc:374] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:57.978703 7828 raft_consensus.cc:729] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fcbd04ca4282b63a4c0908b1b5fd, State: Initialized, Role: FOLLOWER
I20260927 18:32:57.979297 7828 consensus_queue.cc:260] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:57.980372 7830 tablet_bootstrap.cc:654] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:57.981040 7828 ts_tablet_manager.cc:1428] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 18:32:57.981542 7830 log.cc:824] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:57.985277 7831 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Bootstrap starting.
I20260927 18:32:57.988525 7831 tablet_bootstrap.cc:654] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:57.989601 7831 log.cc:824] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Log is configured to *not* fsync() on all Append() calls
I20260927 18:32:57.993121 7830 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: No bootstrap required, opened a new log
I20260927 18:32:57.993377 7831 tablet_bootstrap.cc:492] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: No bootstrap required, opened a new log
I20260927 18:32:57.993392 7830 ts_tablet_manager.cc:1397] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260927 18:32:57.993610 7831 ts_tablet_manager.cc:1397] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:32:58.000442 7830 raft_consensus.cc:348] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.000777 7830 raft_consensus.cc:374] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:58.000890 7830 raft_consensus.cc:729] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfae4e78bf9540788eb6ecd5dc311565, State: Initialized, Role: FOLLOWER
I20260927 18:32:58.000794 7831 raft_consensus.cc:348] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.001211 7831 raft_consensus.cc:374] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:58.001314 7831 raft_consensus.cc:729] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d273b5452b34e25b8d51b42f0271c26, State: Initialized, Role: FOLLOWER
I20260927 18:32:58.001521 7830 consensus_queue.cc:260] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.001891 7831 consensus_queue.cc:260] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.002955 7792 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
I20260927 18:32:58.003407 7830 ts_tablet_manager.cc:1428] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 18:32:58.003690 7651 heartbeater.cc:503] Master 127.0.97.126:35643 was elected leader, sending a full tablet report...
I20260927 18:32:58.004424 7831 ts_tablet_manager.cc:1428] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 18:32:58.074747 7832 raft_consensus.cc:482] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:58.075209 7832 raft_consensus.cc:504] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.076876 7832 leader_election.cc:288] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207), 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
W20260927 18:32:58.083714 7794 tablet.cc:2369] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:32:58.084179 7513 tablet.cc:2369] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:32:58.085427 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.077351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37675 (local address 127.0.97.67:44693)
0927 18:32:58.077996 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.078005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.078139 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:32:58.078493 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.078498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.078577 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.078685 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:58.079499 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.081034 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.083257 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.084268 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.084393 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.084515 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.085077 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.085193 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260927 18:32:58.085606 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.077139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44693 (local address 127.0.97.65:37675)
0927 18:32:58.078008 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:58.078035 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:58.078267 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:58.078720 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:58.078725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:58.078770 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:58.079316 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:58.079328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.081183 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.081186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:58.083061 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:58.083076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.084679 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.084685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:58.084912 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.084938 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:32:58.085043 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:32:58.085167 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":80,"spinlock_wait_cycles":7296,"thread_start_us":477,"threads_started":1}
I20260927 18:32:58.085647 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.077139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39207 (local address 127.0.97.65:39953)
0927 18:32:58.078249 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:58.078264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:58.078377 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:58.078720 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:58.078725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:58.078763 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:58.079316 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:58.079328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.081231 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.081235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:58.083019 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:58.083030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.084735 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.084739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:58.084967 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.084990 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:32:58.085088 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:32:58.085205 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":167,"spinlock_wait_cycles":768,"thread_start_us":774,"threads_started":1}
I20260927 18:32:58.085880 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.077366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39953 (local address 127.0.97.66:39207)
0927 18:32:58.078176 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.078180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.078353 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:32:58.078508 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.078512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.078578 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.078685 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:58.079542 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.081014 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.083164 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.084356 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.084481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.084578 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.085122 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.085267 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
I20260927 18:32:58.086524 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e28ed7a5f7c4ebfb1ded3c8096c0757" candidate_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d273b5452b34e25b8d51b42f0271c26" is_pre_election: true
I20260927 18:32:58.086710 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e28ed7a5f7c4ebfb1ded3c8096c0757" candidate_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfae4e78bf9540788eb6ecd5dc311565" is_pre_election: true
I20260927 18:32:58.087028 7746 raft_consensus.cc:2454] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fcbd04ca4282b63a4c0908b1b5fd in term 0.
I20260927 18:32:58.087200 7605 raft_consensus.cc:2454] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fcbd04ca4282b63a4c0908b1b5fd in term 0.
I20260927 18:32:58.087754 7398 leader_election.cc:302] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d273b5452b34e25b8d51b42f0271c26, 7365fcbd04ca4282b63a4c0908b1b5fd; no voters:
I20260927 18:32:58.088346 7832 raft_consensus.cc:2781] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:58.088507 7832 raft_consensus.cc:482] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:58.088618 7832 raft_consensus.cc:3037] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:58.090988 7832 raft_consensus.cc:504] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.091578 7832 leader_election.cc:288] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [CANDIDATE]: Term 1 election: Requested vote from peers dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207), 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693)
I20260927 18:32:58.091898 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e28ed7a5f7c4ebfb1ded3c8096c0757" candidate_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfae4e78bf9540788eb6ecd5dc311565"
I20260927 18:32:58.092020 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e28ed7a5f7c4ebfb1ded3c8096c0757" candidate_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d273b5452b34e25b8d51b42f0271c26"
I20260927 18:32:58.092133 7605 raft_consensus.cc:3037] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:58.092216 7746 raft_consensus.cc:3037] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:58.094699 7605 raft_consensus.cc:2454] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fcbd04ca4282b63a4c0908b1b5fd in term 1.
I20260927 18:32:58.094756 7746 raft_consensus.cc:2454] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fcbd04ca4282b63a4c0908b1b5fd in term 1.
I20260927 18:32:58.095094 7397 leader_election.cc:302] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fcbd04ca4282b63a4c0908b1b5fd, dfae4e78bf9540788eb6ecd5dc311565; no voters:
I20260927 18:32:58.095441 7832 raft_consensus.cc:2781] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:58.095861 7832 raft_consensus.cc:686] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 LEADER]: Becoming Leader. State: Replica: 7365fcbd04ca4282b63a4c0908b1b5fd, State: Running, Role: LEADER
I20260927 18:32:58.096414 7832 consensus_queue.cc:237] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } }
I20260927 18:32:58.101274 6532 catalog_manager.cc:5495] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65). New cstate: current_term: 1 leader_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:58.139099 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.0.97.65:43871)
0927 18:32:58.134259 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.134265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.134432 (+ 167us) server_negotiation.cc:407] Connection header received
0927 18:32:58.134611 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.134614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.134707 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.134831 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:58.135215 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.135964 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.137235 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.137641 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.137731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.138158 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:58.138649 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:58.138755 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.138824 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.138884 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:32:58.146730 7746 raft_consensus.cc:1261] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Refusing update from remote peer 7365fcbd04ca4282b63a4c0908b1b5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:58.146735 7605 raft_consensus.cc:1261] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Refusing update from remote peer 7365fcbd04ca4282b63a4c0908b1b5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:58.147722 7839 consensus_queue.cc:1035] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:58.148054 7832 consensus_queue.cc:1035] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:58.172849 7844 mvcc.cc:205] Tried to move back new op lower bound from 7334027174479306752 to 7334027174286336000. Current Snapshot: MvccSnapshot[applied={T|T < 7334027174479306752}]
I20260927 18:32:58.177980 6532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:58.178254 6532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:58.194362 6532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c310d59190374de6940119e43dcbc32f: Sending DeleteTablet for 3 replicas of tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757
I20260927 18:32:58.195689 7585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:58 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47904
I20260927 18:32:58.195801 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:58 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:43334
I20260927 18:32:58.197314 7726 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:58 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49940
I20260927 18:32:58.206369 7853 tablet_replica.cc:330] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: stopping tablet replica
I20260927 18:32:58.206933 7853 raft_consensus.cc:2229] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:58.207801 7853 raft_consensus.cc:2258] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:58.209949 7853 ts_tablet_manager.cc:1905] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:58.210083 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6500
I20260927 18:32:58.211122 6562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:32:58.216933 7853 ts_tablet_manager.cc:1918] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:58.217110 7853 log.cc:1196] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/9e28ed7a5f7c4ebfb1ded3c8096c0757
I20260927 18:32:58.217702 7855 tablet_replica.cc:330] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: stopping tablet replica
I20260927 18:32:58.217762 7853 ts_tablet_manager.cc:1939] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting consensus metadata
I20260927 18:32:58.218366 7852 tablet_replica.cc:330] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: stopping tablet replica
I20260927 18:32:58.218375 7855 raft_consensus.cc:2229] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:58.218993 7855 raft_consensus.cc:2258] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:58.218995 7852 raft_consensus.cc:2229] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:58.219585 7852 raft_consensus.cc:2258] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:58.221372 7855 ts_tablet_manager.cc:1905] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:58.221673 7852 ts_tablet_manager.cc:1905] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:58.228322 7852 ts_tablet_manager.cc:1918] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:58.228482 7852 log.cc:1196] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/9e28ed7a5f7c4ebfb1ded3c8096c0757
I20260927 18:32:58.228914 7852 ts_tablet_manager.cc:1939] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P dfae4e78bf9540788eb6ecd5dc311565: Deleting consensus metadata
I20260927 18:32:58.233521 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.197903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.0.97.125:41753)
0927 18:32:58.198114 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.198121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.208333 (+ 10212us) server_negotiation.cc:407] Connection header received
0927 18:32:58.211194 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.211199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.211298 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.211389 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:58.214327 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.215785 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.220556 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.221215 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.221331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.225331 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:58.225338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:58.225342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:58.225384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:58.229873 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.230982 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.230991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.230996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.231181 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.231745 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.231752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.231756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.232169 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:58.232383 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.232951 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.233199 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:32:58.241263 7855 ts_tablet_manager.cc:1918] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:58.241547 7855 log.cc:1196] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/9e28ed7a5f7c4ebfb1ded3c8096c0757
I20260927 18:32:58.242220 7855 ts_tablet_manager.cc:1939] T 9e28ed7a5f7c4ebfb1ded3c8096c0757 P 4d273b5452b34e25b8d51b42f0271c26: Deleting consensus metadata
I20260927 18:32:58.251380 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.208878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.0.97.124:43051)
0927 18:32:58.209059 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.209065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.234549 (+ 25484us) server_negotiation.cc:407] Connection header received
0927 18:32:58.234821 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.234827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.234914 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.235005 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:58.236986 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.238106 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.239993 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.240616 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.240705 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.244301 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:58.244306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:58.244310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:58.244347 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:58.248300 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.249075 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.249080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.249083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.249243 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.249676 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.249683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.249688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.250092 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:58.250332 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.250681 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.250950 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:32:58.369642 6518 catalog_manager.cc:4841] TS 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693): tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 (table test-table [id=67ad541b1944420e8d78256fa3fd146a]) successfully deleted
I20260927 18:32:58.370321 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6500
I20260927 18:32:58.370294 6516 catalog_manager.cc:4841] TS 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871): tablet 9e28ed7a5f7c4ebfb1ded3c8096c0757 (table test-table [id=67ad541b1944420e8d78256fa3fd146a]) successfully deleted
I20260927 18:32:58.419277 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.396895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.97.125:41753)
0927 18:32:58.397070 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.397075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.401266 (+ 4191us) server_negotiation.cc:407] Connection header received
0927 18:32:58.401836 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.401846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.401957 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.402061 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:58.405468 (+ 3407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.406649 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.408843 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.409445 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.409556 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.412986 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:58.412993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:58.412998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:58.413042 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:58.416527 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.417275 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.417280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.417284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.417417 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.417842 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.417847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.417851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.418191 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:58.418401 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.418710 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.418919 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:58.420797 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:58.396443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.0.97.124:43051)
0927 18:32:58.396655 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:58.396665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:58.405369 (+ 8704us) server_negotiation.cc:407] Connection header received
0927 18:32:58.405610 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:58.405615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:58.405690 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:58.405755 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:58.408092 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.409150 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.411171 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:58.411809 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:58.411935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:58.414522 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:58.414527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:58.414530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:58.414558 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:58.417333 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.418083 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.418088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.418098 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.418207 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:58.419211 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:58.419216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:58.419219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:58.419565 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:58.419765 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:58.420195 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:32:58.420397 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:59.172993 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.172142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:37623)
0927 18:32:59.172618 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.172767 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":202,"threads_started":1}
W20260927 18:32:59.173408 7511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.199770 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.199056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:58673)
0927 18:32:59.199523 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.199624 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
W20260927 18:32:59.200222 7651 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.200601 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.199632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:38945)
0927 18:32:59.200292 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.200434 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":463,"threads_started":1}
W20260927 18:32:59.200934 7792 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.689179 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c310d59190374de6940119e43dcbc32f)
I20260927 18:32:59.689502 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } }
I20260927 18:32:59.690871 7867 leader_election.cc:288] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c310d59190374de6940119e43dcbc32f (127.0.97.126:35643), 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124:43051)
I20260927 18:32:59.691601 7868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c310d59190374de6940119e43dcbc32f)
I20260927 18:32:59.691864 7868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } }
I20260927 18:32:59.692421 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.690962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45740)
0927 18:32:59.691886 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.692118 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":769,"threads_started":1}
W20260927 18:32:59.693380 7122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.699800 7868 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c310d59190374de6940119e43dcbc32f (127.0.97.126:35643), 92523705747747c5b96d24aee4daccde (127.0.97.125:41753)
I20260927 18:32:59.702546 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.691061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43051 (local address 127.0.0.1:41928)
0927 18:32:59.692684 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.692707 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.693220 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.694216 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.694225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.694269 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.694585 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.694599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.696665 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.696675 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.699330 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.699349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.701442 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.701452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.701742 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.701819 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.701901 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.702098 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":533,"threads_started":1}
I20260927 18:32:59.704308 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.691378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.97.124:43051)
0927 18:32:59.693741 (+ 2363us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.693749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.693772 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:32:59.693876 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.693932 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.694034 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.694749 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.694979 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.696477 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.699577 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.701207 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.702800 (+ 1593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.702968 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.703029 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.703224 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2239,"thread_start_us":2091,"threads_started":1}
I20260927 18:32:59.705482 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.705062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45768)
0927 18:32:59.705229 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.705319 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 18:32:59.705708 7122 leader_election.cc:334] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c310d59190374de6940119e43dcbc32f (127.0.97.126:35643): Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.705672 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92523705747747c5b96d24aee4daccde" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f1c896d2bf84814b4113c102af6ac1e" is_pre_election: true
I20260927 18:32:59.706084 7336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92523705747747c5b96d24aee4daccde in term 1.
I20260927 18:32:59.706334 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.693763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45752)
0927 18:32:59.705257 (+ 11494us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.705474 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10984,"thread_start_us":11265,"threads_started":1}
W20260927 18:32:59.706633 7311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.707574 7124 leader_election.cc:302] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1c896d2bf84814b4113c102af6ac1e, 92523705747747c5b96d24aee4daccde; no voters: c310d59190374de6940119e43dcbc32f
I20260927 18:32:59.708017 7867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:32:59.708176 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Starting leader election (detected failure of leader c310d59190374de6940119e43dcbc32f)
I20260927 18:32:59.708285 7867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:32:59.708323 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.707910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45782)
0927 18:32:59.708107 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.708186 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 18:32:59.708503 7311 leader_election.cc:334] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c310d59190374de6940119e43dcbc32f (127.0.97.126:35643): Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.711174 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } }
I20260927 18:32:59.711871 7867 leader_election.cc:288] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 election: Requested vote from peers c310d59190374de6940119e43dcbc32f (127.0.97.126:35643), 3f1c896d2bf84814b4113c102af6ac1e (127.0.97.124:43051)
I20260927 18:32:59.712445 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.712036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45788)
0927 18:32:59.712198 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.712282 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:32:59.712391 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92523705747747c5b96d24aee4daccde" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f1c896d2bf84814b4113c102af6ac1e"
I20260927 18:32:59.712538 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.693842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41753 (local address 127.0.0.1:59686)
0927 18:32:59.705291 (+ 11449us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.705308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.705472 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.706139 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.706143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.706191 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.706690 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.706698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.708169 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.708176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.710260 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.710274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.711689 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.711696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.711912 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.711949 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.712043 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.712185 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11330,"thread_start_us":10793,"threads_started":1}
I20260927 18:32:59.712656 7336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:32:59.712957 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.694512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.97.125:41753)
0927 18:32:59.704976 (+ 10464us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.704983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.705590 (+ 607us) server_negotiation.cc:407] Connection header received
0927 18:32:59.705649 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.705654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.705733 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.705867 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.706877 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.707990 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.710506 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.711521 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.711632 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.711771 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.712311 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.712481 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10331,"thread_start_us":10479,"threads_started":1}
I20260927 18:32:59.713595 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.713155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45798)
0927 18:32:59.713341 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.713432 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:32:59.713510 7147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f1c896d2bf84814b4113c102af6ac1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92523705747747c5b96d24aee4daccde" is_pre_election: true
W20260927 18:32:59.713831 7122 leader_election.cc:334] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c310d59190374de6940119e43dcbc32f (127.0.97.126:35643): Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
I20260927 18:32:59.713920 7147 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f1c896d2bf84814b4113c102af6ac1e in current term 2: Already voted for candidate 92523705747747c5b96d24aee4daccde in this term.
I20260927 18:32:59.714634 7312 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f1c896d2bf84814b4113c102af6ac1e; no voters: 92523705747747c5b96d24aee4daccde, c310d59190374de6940119e43dcbc32f
I20260927 18:32:59.715675 7336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92523705747747c5b96d24aee4daccde in term 2.
I20260927 18:32:59.715855 7868 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:32:59.716115 7124 leader_election.cc:302] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1c896d2bf84814b4113c102af6ac1e, 92523705747747c5b96d24aee4daccde; no voters: c310d59190374de6940119e43dcbc32f
I20260927 18:32:59.716365 7867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:32:59.716818 7867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [term 2 LEADER]: Becoming Leader. State: Replica: 92523705747747c5b96d24aee4daccde, State: Running, Role: LEADER
I20260927 18:32:59.717211 7867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } }
I20260927 18:32:59.718766 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92523705747747c5b96d24aee4daccde. Latest consensus state: current_term: 2 leader_uuid: "92523705747747c5b96d24aee4daccde" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:59.719091 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:59.719671 7879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:32:59.723044 7879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:32:59.723846 7879 catalog_manager.cc:1259] Loaded cluster ID: b7516ade86d64b1f93f405d4f89955e4
I20260927 18:32:59.723940 7879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:32:59.725103 7879 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:32:59.725925 7879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Loaded TSK: 0
I20260927 18:32:59.726727 7879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:32:59.745266 7136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:32:59.752372 7136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:32:59.756433 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.756074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45816)
0927 18:32:59.756238 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.756311 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:32:59.756564 7336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [term 2 FOLLOWER]: Refusing update from remote peer 92523705747747c5b96d24aee4daccde: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 18:32:59.757315 7877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 18:32:59.757544 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.757063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45824)
0927 18:32:59.757267 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.757398 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 18:32:59.757710 7122 consensus_peers.cc:487] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde -> Peer c310d59190374de6940119e43dcbc32f (127.0.97.126:35643): Couldn't send request to peer c310d59190374de6940119e43dcbc32f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:32:59.759872 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92523705747747c5b96d24aee4daccde. Latest consensus state: current_term: 2 leader_uuid: "92523705747747c5b96d24aee4daccde" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:59.760223 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:59.761684 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "92523705747747c5b96d24aee4daccde" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:59.762055 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde [sys.catalog]: This master's current role is: LEADER
I20260927 18:32:59.762415 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "92523705747747c5b96d24aee4daccde" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c310d59190374de6940119e43dcbc32f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35643 } } peers { permanent_uuid: "92523705747747c5b96d24aee4daccde" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41753 } attrs { promote: false } } peers { permanent_uuid: "3f1c896d2bf84814b4113c102af6ac1e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43051 } attrs { promote: false } } }
I20260927 18:32:59.762696 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1c896d2bf84814b4113c102af6ac1e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:32:59.777904 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.770611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44693 (local address 127.0.0.1:49944)
0927 18:32:59.770796 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.770837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.771019 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.772203 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.772207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.772231 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.772426 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.772434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.773724 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.773730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.775656 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.775669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.777097 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.777103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.777299 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.777333 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.777428 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.777565 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:32:59.777953 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.770666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.97.67:44693)
0927 18:32:59.771920 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.771925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.771940 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:32:59.771994 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.771998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.772074 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.772186 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.772563 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.773598 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.775819 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.776953 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.777077 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.777185 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.777494 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.777629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":1062,"threads_started":1}
I20260927 18:32:59.778744 7726 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (DEFAULT_TABLE table=test-table [id=262e717bb9d949d08c932a13f7f3e4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:59.779206 7726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:59.779992 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.771582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39207 (local address 127.0.0.1:47918)
0927 18:32:59.772196 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.772217 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.772329 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.773160 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.773165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.773193 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.773625 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.773640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.775150 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.775156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.777566 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.777580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.779170 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.779179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.779342 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.779373 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.779461 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.779575 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1,"thread_start_us":1364,"threads_started":1}
I20260927 18:32:59.780689 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.772396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.0.97.66:39207)
0927 18:32:59.772837 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.772842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.772859 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:32:59.772921 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.772925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.772993 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.773822 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.773994 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.775019 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.777735 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.778806 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.778924 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.779032 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.780271 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.780413 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":245,"threads_started":1}
I20260927 18:32:59.781447 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.770980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43871 (local address 127.0.0.1:43360)
0927 18:32:59.771413 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.771429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.771546 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.774411 (+ 2865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.774416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.774441 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.774823 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.774833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.776286 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.776291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.778323 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.778335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.780757 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.780763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.780897 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.780918 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.780972 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.781062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2}
I20260927 18:32:59.781751 7585 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (DEFAULT_TABLE table=test-table [id=262e717bb9d949d08c932a13f7f3e4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:59.782176 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:59.782320 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.772001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.97.65:43871)
0927 18:32:59.774123 (+ 2122us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.774131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.774149 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:32:59.774212 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.774217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.774285 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.774381 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.774993 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.776131 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.779260 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.780521 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.781740 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.781893 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.781945 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.782078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2003,"thread_start_us":1889,"threads_started":1}
I20260927 18:32:59.782938 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (DEFAULT_TABLE table=test-table [id=262e717bb9d949d08c932a13f7f3e4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:32:59.783306 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:32:59.785799 7890 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Bootstrap starting.
I20260927 18:32:59.787591 7890 tablet_bootstrap.cc:654] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:59.788547 7891 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Bootstrap starting.
I20260927 18:32:59.790081 7891 tablet_bootstrap.cc:654] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:59.792021 7891 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: No bootstrap required, opened a new log
I20260927 18:32:59.792196 7891 ts_tablet_manager.cc:1397] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:32:59.792862 7891 raft_consensus.cc:348] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.793079 7891 raft_consensus.cc:374] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:59.793164 7891 raft_consensus.cc:729] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfae4e78bf9540788eb6ecd5dc311565, State: Initialized, Role: FOLLOWER
I20260927 18:32:59.793417 7891 consensus_queue.cc:260] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.794299 7891 ts_tablet_manager.cc:1428] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:32:59.797992 7652 heartbeater.cc:503] Master 127.0.97.125:41753 was elected leader, sending a full tablet report...
I20260927 18:32:59.802536 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.794990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:60027)
0927 18:32:59.802208 (+ 7218us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.802324 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7116,"thread_start_us":7004,"threads_started":1}
I20260927 18:32:59.803525 7893 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Bootstrap starting.
I20260927 18:32:59.803915 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.803320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:58025)
0927 18:32:59.803734 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.803811 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260927 18:32:59.804921 7890 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: No bootstrap required, opened a new log
I20260927 18:32:59.805086 7893 tablet_bootstrap.cc:654] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Neither blocks nor log segments found. Creating new log.
I20260927 18:32:59.805114 7890 ts_tablet_manager.cc:1397] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20260927 18:32:59.805770 7890 raft_consensus.cc:348] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.806119 7890 raft_consensus.cc:374] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:59.806242 7890 raft_consensus.cc:729] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d273b5452b34e25b8d51b42f0271c26, State: Initialized, Role: FOLLOWER
I20260927 18:32:59.806516 7890 consensus_queue.cc:260] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.807206 7893 tablet_bootstrap.cc:492] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: No bootstrap required, opened a new log
I20260927 18:32:59.807348 7893 ts_tablet_manager.cc:1397] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:32:59.807361 7890 ts_tablet_manager.cc:1428] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:32:59.808187 7893 raft_consensus.cc:348] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.808449 7893 raft_consensus.cc:374] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:32:59.808561 7893 raft_consensus.cc:729] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fcbd04ca4282b63a4c0908b1b5fd, State: Initialized, Role: FOLLOWER
I20260927 18:32:59.808830 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.807605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:48951)
0927 18:32:59.808211 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.808348 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260927 18:32:59.808787 7893 consensus_queue.cc:260] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.810313 7893 ts_tablet_manager.cc:1428] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:32:59.810861 7793 heartbeater.cc:503] Master 127.0.97.125:41753 was elected leader, sending a full tablet report...
I20260927 18:32:59.811816 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.810940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:58117)
0927 18:32:59.811430 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.811568 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"threads_started":1}
I20260927 18:32:59.812794 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.809663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:52083)
0927 18:32:59.812574 (+ 2911us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.812654 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2816}
I20260927 18:32:59.813405 7512 heartbeater.cc:503] Master 127.0.97.125:41753 was elected leader, sending a full tablet report...
I20260927 18:32:59.813992 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.813623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:49565)
0927 18:32:59.813757 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.813860 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 18:32:59.838109 7898 raft_consensus.cc:482] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:32:59.838341 7898 raft_consensus.cc:504] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.839886 7898 leader_election.cc:288] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871), dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:59.846554 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.840092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43871 (local address 127.0.97.67:55499)
0927 18:32:59.840347 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.840367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.840564 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.840978 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.840984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.841039 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.841303 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.841317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.842498 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.842502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.844415 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.844429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.845859 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.845864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.845976 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.846040 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.846135 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.846248 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":1,"spinlock_wait_cycles":22528}
I20260927 18:32:59.846577 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.840173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55499 (local address 127.0.97.65:43871)
0927 18:32:59.840391 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.840400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.840428 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:32:59.840734 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.840739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.840834 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.840919 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.841458 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.842359 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.844558 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.845624 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.845764 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.845901 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.846188 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.846320 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:32:59.847350 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.840278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55509 (local address 127.0.97.66:39207)
0927 18:32:59.840470 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.840477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.840998 (+ 521us) server_negotiation.cc:407] Connection header received
0927 18:32:59.841169 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.841173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.841235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.841304 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:32:59.842028 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.842876 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.844961 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.846307 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.846434 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.846531 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.846891 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.847050 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:32:59.847353 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9f850c2c04d18b8f3e9c8c9069ecd" candidate_uuid: "4d273b5452b34e25b8d51b42f0271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" is_pre_election: true
I20260927 18:32:59.847817 7465 raft_consensus.cc:2454] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d273b5452b34e25b8d51b42f0271c26 in term 0.
I20260927 18:32:59.848016 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.840160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39207 (local address 127.0.97.67:55509)
0927 18:32:59.840872 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.840885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:32:59.841042 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:32:59.841341 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:32:59.841347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:32:59.841621 (+ 274us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:32:59.841882 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.841891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.843022 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.843027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.844812 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:32:59.844821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.846488 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.846494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:32:59.846703 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.846739 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:32:59.847525 (+ 786us) client_negotiation.cc:240] Negotiation successful
0927 18:32:59.847662 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"spinlock_wait_cycles":3328,"thread_start_us":493,"threads_started":1}
I20260927 18:32:59.848459 7679 leader_election.cc:302] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d273b5452b34e25b8d51b42f0271c26, 7365fcbd04ca4282b63a4c0908b1b5fd; no voters:
I20260927 18:32:59.848635 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9f850c2c04d18b8f3e9c8c9069ecd" candidate_uuid: "4d273b5452b34e25b8d51b42f0271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfae4e78bf9540788eb6ecd5dc311565" is_pre_election: true
I20260927 18:32:59.848886 7898 raft_consensus.cc:2781] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:32:59.848889 7605 raft_consensus.cc:2454] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d273b5452b34e25b8d51b42f0271c26 in term 0.
I20260927 18:32:59.849026 7898 raft_consensus.cc:482] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:32:59.849141 7898 raft_consensus.cc:3037] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:59.851289 7898 raft_consensus.cc:504] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.851908 7898 leader_election.cc:288] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871), dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207)
I20260927 18:32:59.852309 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9f850c2c04d18b8f3e9c8c9069ecd" candidate_uuid: "4d273b5452b34e25b8d51b42f0271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd"
I20260927 18:32:59.852419 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9f850c2c04d18b8f3e9c8c9069ecd" candidate_uuid: "4d273b5452b34e25b8d51b42f0271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfae4e78bf9540788eb6ecd5dc311565"
I20260927 18:32:59.852522 7465 raft_consensus.cc:3037] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:59.852624 7605 raft_consensus.cc:3037] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:32:59.855521 7605 raft_consensus.cc:2454] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d273b5452b34e25b8d51b42f0271c26 in term 1.
I20260927 18:32:59.855520 7465 raft_consensus.cc:2454] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d273b5452b34e25b8d51b42f0271c26 in term 1.
I20260927 18:32:59.855901 7679 leader_election.cc:302] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d273b5452b34e25b8d51b42f0271c26, 7365fcbd04ca4282b63a4c0908b1b5fd; no voters:
I20260927 18:32:59.856175 7898 raft_consensus.cc:2781] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:32:59.856604 7898 raft_consensus.cc:686] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 1 LEADER]: Becoming Leader. State: Replica: 4d273b5452b34e25b8d51b42f0271c26, State: Running, Role: LEADER
I20260927 18:32:59.857010 7898 consensus_queue.cc:237] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } }
I20260927 18:32:59.857586 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.857181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:45385)
0927 18:32:59.857357 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.857459 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:32:59.860610 7137 catalog_manager.cc:5495] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "4d273b5452b34e25b8d51b42f0271c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d273b5452b34e25b8d51b42f0271c26" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 } health_report { overall_health: UNKNOWN } } }
I20260927 18:32:59.883822 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.879112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.0.97.67:44693)
0927 18:32:59.879292 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.879296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.879312 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:32:59.879394 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.879397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.879519 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.879640 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:32:59.879964 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.880683 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.881914 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.882321 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.882410 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.882816 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:32:59.883378 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:32:59.883536 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.883573 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.883633 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:32:59.887949 7605 raft_consensus.cc:1261] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Refusing update from remote peer 4d273b5452b34e25b8d51b42f0271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:59.888352 7465 raft_consensus.cc:1261] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Refusing update from remote peer 4d273b5452b34e25b8d51b42f0271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:32:59.888801 7903 consensus_queue.cc:1035] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfae4e78bf9540788eb6ecd5dc311565" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:59.889241 7898 consensus_queue.cc:1035] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fcbd04ca4282b63a4c0908b1b5fd" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:32:59.892204 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.891527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:49289)
0927 18:32:59.891791 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.891900 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 18:32:59.892439 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.891700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:33565)
0927 18:32:59.892095 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.892237 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260927 18:32:59.897997 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.897401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.65:50395)
0927 18:32:59.897659 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.897739 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 18:32:59.902081 7908 mvcc.cc:205] Tried to move back new op lower bound from 7334027181613219840 to 7334027181498703872. Current Snapshot: MvccSnapshot[applied={T|T < 7334027181613219840}]
I20260927 18:32:59.903352 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.902773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.66:44513)
0927 18:32:59.903054 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.903160 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 18:32:59.905236 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.904232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:45021)
0927 18:32:59.904978 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.905081 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587}
I20260927 18:32:59.908365 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.907946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.97.67:56221)
0927 18:32:59.908143 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:32:59.908248 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 18:32:59.917461 7137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:59.917809 7137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:32:59.923241 7137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92523705747747c5b96d24aee4daccde: Sending DeleteTablet for 3 replicas of tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd
I20260927 18:32:59.924259 7726 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:59 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:49944
I20260927 18:32:59.924513 7585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:59 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:47918
I20260927 18:32:59.924685 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:32:59 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:43360
I20260927 18:32:59.924994 7915 tablet_replica.cc:330] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: stopping tablet replica
I20260927 18:32:59.925172 7915 raft_consensus.cc:2229] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:32:59.925727 7915 raft_consensus.cc:2258] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:59.926795 7915 ts_tablet_manager.cc:1905] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:59.927037 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 7303
I20260927 18:32:59.933686 7915 ts_tablet_manager.cc:1918] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:59.933848 7915 log.cc:1196] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/9ec9f850c2c04d18b8f3e9c8c9069ecd
I20260927 18:32:59.934382 7915 ts_tablet_manager.cc:1939] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 4d273b5452b34e25b8d51b42f0271c26: Deleting consensus metadata
I20260927 18:32:59.935626 7123 catalog_manager.cc:4841] TS 4d273b5452b34e25b8d51b42f0271c26 (127.0.97.67:44693): tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (table test-table [id=262e717bb9d949d08c932a13f7f3e4d8]) successfully deleted
I20260927 18:32:59.938117 7917 tablet_replica.cc:330] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: stopping tablet replica
I20260927 18:32:59.938243 7916 tablet_replica.cc:330] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: stopping tablet replica
I20260927 18:32:59.938448 7916 raft_consensus.cc:2229] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:59.938552 7917 raft_consensus.cc:2229] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:32:59.939018 7917 raft_consensus.cc:2258] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:59.939119 7916 raft_consensus.cc:2258] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:32:59.940447 7916 ts_tablet_manager.cc:1905] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:59.940436 7917 ts_tablet_manager.cc:1905] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:32:59.946956 7916 ts_tablet_manager.cc:1918] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:59.947080 7916 log.cc:1196] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/9ec9f850c2c04d18b8f3e9c8c9069ecd
I20260927 18:32:59.947491 7916 ts_tablet_manager.cc:1939] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P dfae4e78bf9540788eb6ecd5dc311565: Deleting consensus metadata
I20260927 18:32:59.947814 7917 ts_tablet_manager.cc:1918] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:32:59.947996 7917 log.cc:1196] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/9ec9f850c2c04d18b8f3e9c8c9069ecd
I20260927 18:32:59.948460 7917 ts_tablet_manager.cc:1939] T 9ec9f850c2c04d18b8f3e9c8c9069ecd P 7365fcbd04ca4282b63a4c0908b1b5fd: Deleting consensus metadata
I20260927 18:32:59.948616 7122 catalog_manager.cc:4841] TS dfae4e78bf9540788eb6ecd5dc311565 (127.0.97.66:39207): tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (table test-table [id=262e717bb9d949d08c932a13f7f3e4d8]) successfully deleted
I20260927 18:32:59.949535 7121 catalog_manager.cc:4841] TS 7365fcbd04ca4282b63a4c0908b1b5fd (127.0.97.65:43871): tablet 9ec9f850c2c04d18b8f3e9c8c9069ecd (table test-table [id=262e717bb9d949d08c932a13f7f3e4d8]) successfully deleted
I20260927 18:32:59.959723 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:32:59.928094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.65:43871)
0927 18:32:59.928263 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:32:59.928269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:32:59.943901 (+ 15632us) server_negotiation.cc:407] Connection header received
0927 18:32:59.944324 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:32:59.944331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:32:59.944416 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:32:59.944500 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:32:59.946543 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.947653 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.949582 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:32:59.950233 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:32:59.950362 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:32:59.952825 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:32:59.952842 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:32:59.952863 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:32:59.952904 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:32:59.956808 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:59.957476 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:59.957487 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:59.957490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:59.957634 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:32:59.958126 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:32:59.958131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:32:59.958134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:32:59.958582 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:32:59.958859 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:32:59.959167 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 18:32:59.959375 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:32:59.960438 7505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:00.078078 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 7303
I20260927 18:33:00.110129 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 7522
I20260927 18:33:00.128450 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:00.110863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.97.66:39207)
0927 18:33:00.111045 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:00.111053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:00.111227 (+ 174us) server_negotiation.cc:407] Connection header received
0927 18:33:00.111446 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:00.111450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:00.111519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:00.111631 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:00.112842 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:00.113747 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:00.115031 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:00.115419 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:00.115546 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:00.122685 (+ 7139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:00.122701 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:00.122716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:00.122750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:00.125633 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:00.126388 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:00.126397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:00.126399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:00.126534 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:00.126913 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:00.126920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:00.126923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:00.127282 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:00.127615 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:00.127874 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 18:33:00.128082 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:33:00.129216 7645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:00.249552 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:00.249018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45840)
0927 18:33:00.249250 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:00.249362 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:33:00.250801 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:00.250364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35643 (local address 127.0.0.1:45846)
0927 18:33:00.250536 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:00.250637 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:33:00.257516 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 7522
I20260927 18:33:00.276746 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 7663
I20260927 18:33:00.293444 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:00.277565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.0.97.67:44693)
0927 18:33:00.277748 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:00.277756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:00.277826 (+ 70us) server_negotiation.cc:407] Connection header received
0927 18:33:00.278034 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:00.278040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:00.278125 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:00.278234 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:00.279727 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:00.280931 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:00.282363 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:00.283101 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:00.283239 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:00.285913 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:00.285940 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:00.285961 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:00.286009 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:00.290302 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:00.291064 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:00.291077 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:00.291080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:00.291281 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:00.291711 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:00.291716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:00.291720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:00.292199 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:00.292503 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:00.292777 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 18:33:00.293002 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:33:00.294399 7786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:00.422374 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 7663
I20260927 18:33:00.443373 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 6993
I20260927 18:33:00.444486 7167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:00.565114 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 6993
W20260927 18:33:00.585732 7311 connection.cc:535] server connection from 127.0.0.1:41928 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:33:00.586135 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 7091
I20260927 18:33:00.587189 7356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:00.701514 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 7091
2026-09-27T18:33:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9503 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:33:00 dist-test-slave-n3p4 krb5kdc[7926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:33:00 dist-test-slave-n3p4 krb5kdc[7926](info): set up 2 sockets
Sep 27 18:33:00 dist-test-slave-n3p4 krb5kdc[7926](info): commencing operation
krb5kdc: starting...
W20260927 18:33:01.802970 389 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:33:01 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260927 18:33:02.019678 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34859
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:02.340900 7939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:02.341185 7939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:02.341337 7939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:02.348567 7939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:02.348663 7939 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:33:02.348744 7939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:02.348852 7939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:02.348964 7939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:02.349074 7939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:02.359197 7939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34859
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:02.360504 7939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:02.361887 7939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:02.373847 7944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:02.373903 7947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:02.373816 7945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:02.374940 7939 server_base.cc:1034] running on GCE node
I20260927 18:33:02.375739 7939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:02.377086 7939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:02.378345 7939 hybrid_clock.cc:648] HybridClock initialized: now 1790533982378276 us; error 46 us; skew 500 ppm
I20260927 18:33:02.378684 7939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:02 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:02.381716 7939 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260927 18:33:02.383157 7939 webserver.cc:460] Webserver started at http://127.0.97.126:34135/ using document root <none> and password file <none>
I20260927 18:33:02.383843 7939 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:02.383970 7939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:02.384279 7939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:02.387349 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "353e13b858534deb998de4ea68271f71"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "5438411ab569a7b35d2cd2f167cdf513"
server_key_iv: "dfcf8db3d3a269efb6e846823a590ccf"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.388125 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "353e13b858534deb998de4ea68271f71"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "5438411ab569a7b35d2cd2f167cdf513"
server_key_iv: "dfcf8db3d3a269efb6e846823a590ccf"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.392978 7939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 18:33:02.396817 7954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:02.398185 7939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:33:02.398378 7939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "353e13b858534deb998de4ea68271f71"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "5438411ab569a7b35d2cd2f167cdf513"
server_key_iv: "dfcf8db3d3a269efb6e846823a590ccf"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.398620 7939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:02.416101 7939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:02.420917 7939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:02.421208 7939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:02.444024 7939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34859
I20260927 18:33:02.444058 8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34859 every 8 connection(s)
I20260927 18:33:02.445422 7939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:02.449234 8016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:02.450052 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 7939
I20260927 18:33:02.450284 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:33:02.451057 389 external_mini_cluster.cc:1428] Setting key 7e126b309f438d997706f8db4de7df39
I20260927 18:33:02.463590 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Bootstrap starting.
Sep 27 18:33:02 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:02.467348 8016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:02.468703 8016 log.cc:824] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:02.472147 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: No bootstrap required, opened a new log
I20260927 18:33:02.476019 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:02.453151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.0.97.126:34859)
0927 18:33:02.454036 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:02.454048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:02.454096 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:33:02.455824 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:02.455849 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:02.456285 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:02.456653 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:02.458485 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:02.459953 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:02.461770 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:02.462551 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:02.462692 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:02.467355 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:02.467367 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:02.467388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:02.467439 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:02.471199 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:02.472011 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:02.472021 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:02.472038 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:02.472262 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:02.472677 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:02.472682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:02.472685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:02.473391 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:02.473689 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:02.474239 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 18:33:02.474667 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":435,"threads_started":1}
I20260927 18:33:02.481282 8016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } }
I20260927 18:33:02.481596 8016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:02.481678 8016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353e13b858534deb998de4ea68271f71, State: Initialized, Role: FOLLOWER
I20260927 18:33:02.482213 8016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } }
I20260927 18:33:02.482430 8016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:02.482565 8016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:02.482738 8016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:02.484851 8016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } }
I20260927 18:33:02.485342 8016 leader_election.cc:302] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353e13b858534deb998de4ea68271f71; no voters:
I20260927 18:33:02.485944 8016 leader_election.cc:288] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:33:02.486099 8021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:02.486835 8021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [term 1 LEADER]: Becoming Leader. State: Replica: 353e13b858534deb998de4ea68271f71, State: Running, Role: LEADER
I20260927 18:33:02.487377 8021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } }
I20260927 18:33:02.488149 8016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:02.490586 8022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353e13b858534deb998de4ea68271f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } } }
I20260927 18:33:02.490579 8023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353e13b858534deb998de4ea68271f71. Latest consensus state: current_term: 1 leader_uuid: "353e13b858534deb998de4ea68271f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e13b858534deb998de4ea68271f71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34859 } } }
I20260927 18:33:02.490999 8022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:02.491006 8023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:02.492594 8030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:02.497346 8030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:02.505982 8030 catalog_manager.cc:1347] Generated new cluster ID: 9169dae63d484331b5b474e65c729de7
I20260927 18:33:02.506101 8030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:02.537122 8030 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:33:02.539078 8030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:02.547958 8030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Generated new TSK 0
I20260927 18:33:02.548806 8030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260927 18:33:02.607676 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34859
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:02.932555 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:02.932821 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:02.932915 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:02.940315 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:02.940424 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:02.940510 8042 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:33:02.940583 8042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:02.950165 8042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:34859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:02.951327 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:02.952523 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:02.965515 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:02.965739 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:02.965764 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:02.966737 8042 server_base.cc:1034] running on GCE node
I20260927 18:33:02.967321 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:02.968220 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:02.969439 8042 hybrid_clock.cc:648] HybridClock initialized: now 1790533982969356 us; error 55 us; skew 500 ppm
I20260927 18:33:02.969753 8042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:02 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:02.972849 8042 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260927 18:33:02.974437 8042 webserver.cc:460] Webserver started at http://127.0.97.65:41275/ using document root <none> and password file <none>
I20260927 18:33:02.975294 8042 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:02.975381 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:02.975630 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:02.978807 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "45de60c2741043f9848d383bbe637ba3"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "866b3445faae29dbabaf69a7a0555bd5"
server_key_iv: "0d3544b564ecaed36a7486c5868cd6ff"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.979663 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "45de60c2741043f9848d383bbe637ba3"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "866b3445faae29dbabaf69a7a0555bd5"
server_key_iv: "0d3544b564ecaed36a7486c5868cd6ff"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.985211 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 18:33:02.988991 8057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:02.990382 8042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:02.990568 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "45de60c2741043f9848d383bbe637ba3"
format_stamp: "Formatted at 2026-09-27 18:33:02 on dist-test-slave-n3p4"
server_key: "866b3445faae29dbabaf69a7a0555bd5"
server_key_iv: "0d3544b564ecaed36a7486c5868cd6ff"
server_key_version: "encryptionkey@0"
I20260927 18:33:02.990768 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:03.012964 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:03.017264 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:03.017556 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:03.020525 8042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:03.020637 8042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:03.020797 8042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:03.020943 8042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 18:33:03 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:03.044502 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.021014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53345 (local address 127.0.97.126:34859)
0927 18:33:03.021818 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:03.021826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:03.022870 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 18:33:03.025991 (+ 3121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:03.026000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:03.026092 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:03.026230 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:03.028998 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.029988 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.031573 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.031980 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.032110 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.036724 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:03.036735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:03.036744 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:03.036801 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:03.040606 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.041656 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.041666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.041672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.041889 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.042443 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.042450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.042454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.042859 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:03.043028 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:03.043922 (+ 894us) server_negotiation.cc:299] Negotiation successful
0927 18:33:03.044130 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":551,"threads_started":1}
I20260927 18:33:03.045485 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.021699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.65:53345)
0927 18:33:03.022670 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:03.022725 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:03.025616 (+ 2891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:03.026614 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:03.026629 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:03.027622 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:03.028756 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.028792 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.030150 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.030160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.031415 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.031426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.032228 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.032235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.032441 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.033212 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:03.033255 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:03.036472 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:03.040796 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.040808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.040825 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.041491 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.042031 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.042042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.042046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.042320 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.043015 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:03.043023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:03.043521 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 18:33:03.043848 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 18:33:03.044365 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":352,"threads_started":1}
I20260927 18:33:03.067903 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.052287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47917 (local address 127.0.97.126:34859)
0927 18:33:03.052436 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:03.052468 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:03.053157 (+ 689us) server_negotiation.cc:407] Connection header received
0927 18:33:03.053362 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:03.053365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:03.053420 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:03.053551 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:03.054900 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.055988 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.058071 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.058599 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.058727 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.061626 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:03.061631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:03.061634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:03.061662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:03.064843 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.065824 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.065831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.065837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.065966 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.066353 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.066361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.066365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.066626 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:03.066770 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:03.067340 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 18:33:03.067582 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:33:03.067966 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.052380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.65:47917)
0927 18:33:03.053056 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:03.053074 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:03.053221 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:03.053572 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:03.053576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:03.054154 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:03.054751 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.054767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.056116 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.056122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.057856 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.057869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.058853 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.058864 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.059149 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.059900 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:03.059937 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:03.061379 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:03.065025 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.065031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.065035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.065662 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.066087 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.066092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.066096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.066241 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.066749 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:03.066753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:03.066977 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 18:33:03.067147 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:33:03.067491 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":406,"threads_started":1}
I20260927 18:33:03.079384 8042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36073
I20260927 18:33:03.079407 8177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36073 every 8 connection(s)
I20260927 18:33:03.080716 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:03.083005 8178 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34859
I20260927 18:33:03.083249 8178 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:03.083947 8178 heartbeater.cc:511] Master 127.0.97.126:34859 requested a full tablet report, sending...
I20260927 18:33:03.085929 7971 ts_manager.cc:194] Registered new tserver with Master: 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073)
I20260927 18:33:03.087347 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:53345
I20260927 18:33:03.087947 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8042
I20260927 18:33:03.088167 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:33:03.088900 389 external_mini_cluster.cc:1428] Setting key ac411e6fd08403f18185438d8a7f71ff
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260927 18:33:03.131279 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34859
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:03.454416 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:03.454679 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:03.454855 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:03.462389 8182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:03.462503 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:03.462692 8182 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:33:03.462846 8182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:03.472976 8182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:34859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:03.474356 8182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:03.475821 8182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:03.490639 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:03.490769 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:03.491118 8190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:03.491911 8182 server_base.cc:1034] running on GCE node
I20260927 18:33:03.492564 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:03.493592 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:03.494830 8182 hybrid_clock.cc:648] HybridClock initialized: now 1790533983494763 us; error 44 us; skew 500 ppm
I20260927 18:33:03.495141 8182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:03 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:03.498265 8182 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260927 18:33:03.499688 8182 webserver.cc:460] Webserver started at http://127.0.97.66:40589/ using document root <none> and password file <none>
I20260927 18:33:03.500383 8182 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:03.500505 8182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:03.500819 8182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:03.503798 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "413d0204bde24189839d3eb9ae6cae3a"
format_stamp: "Formatted at 2026-09-27 18:33:03 on dist-test-slave-n3p4"
server_key: "136681aa3b61309e41d648422cc95f41"
server_key_iv: "9cc4934105ded68412dfcf641b004e2b"
server_key_version: "encryptionkey@0"
I20260927 18:33:03.504556 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "413d0204bde24189839d3eb9ae6cae3a"
format_stamp: "Formatted at 2026-09-27 18:33:03 on dist-test-slave-n3p4"
server_key: "136681aa3b61309e41d648422cc95f41"
server_key_iv: "9cc4934105ded68412dfcf641b004e2b"
server_key_version: "encryptionkey@0"
I20260927 18:33:03.509352 8182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 18:33:03.513199 8197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:03.514551 8182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:33:03.514719 8182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "413d0204bde24189839d3eb9ae6cae3a"
format_stamp: "Formatted at 2026-09-27 18:33:03 on dist-test-slave-n3p4"
server_key: "136681aa3b61309e41d648422cc95f41"
server_key_iv: "9cc4934105ded68412dfcf641b004e2b"
server_key_version: "encryptionkey@0"
I20260927 18:33:03.514925 8182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:03.541869 8182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:03.546283 8182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:03.546550 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:03.549265 8182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:03.549371 8182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:03.549512 8182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:03.549588 8182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:33:03 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:03.574488 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.550108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:38577 (local address 127.0.97.126:34859)
0927 18:33:03.550271 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:03.550276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:03.551701 (+ 1425us) server_negotiation.cc:407] Connection header received
0927 18:33:03.553965 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:03.553971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:03.554074 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:03.554193 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:03.556784 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.557811 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.560217 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.560695 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.560845 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.565677 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:03.565686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:03.565692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:03.565739 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:03.570052 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.571177 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.571186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.571193 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.571383 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.572023 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.572028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.572032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.572438 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:03.572629 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:03.573651 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0927 18:33:03.573973 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:33:03.575292 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.550671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.66:38577)
0927 18:33:03.551522 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:03.551574 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:03.553570 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:03.554517 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:03.554537 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:03.555519 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:03.556566 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.556597 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.558040 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.558051 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.560064 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.560082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.560949 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.560965 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.561181 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.562139 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:03.562177 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:03.565342 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:03.570216 (+ 4874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.570230 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.570252 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.571022 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.571531 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.571539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.571544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.571830 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.572604 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:03.572617 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:03.573146 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 18:33:03.573509 (+ 363us) client_negotiation.cc:240] Negotiation successful
0927 18:33:03.574107 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":368,"threads_started":1}
I20260927 18:33:03.598168 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.582320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58061 (local address 127.0.97.126:34859)
0927 18:33:03.582533 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:03.582541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:03.583194 (+ 653us) server_negotiation.cc:407] Connection header received
0927 18:33:03.583346 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:03.583353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:03.583453 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:03.583585 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:03.584448 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.585988 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.587291 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.588048 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.588197 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.590978 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:03.590986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:03.590995 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:03.591025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:03.594613 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.595770 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.595779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.595785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.595962 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:03.596479 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:03.596485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:03.596489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:03.596988 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:03.597142 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:03.597559 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 18:33:03.597772 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:33:03.598290 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:03.582342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.66:58061)
0927 18:33:03.583066 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:03.583090 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:03.583228 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:03.583617 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:03.583620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:03.584035 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:03.584307 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.584315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.586137 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.586152 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.587157 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:03.587168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:03.588311 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:03.588322 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:03.588603 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:03.589435 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:03.589466 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:03.590727 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:03.594772 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.594781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.594788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.595582 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.596103 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:03.596110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:03.596116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:03.596313 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:03.597115 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:03.597120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:03.597307 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 18:33:03.597465 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 18:33:03.597815 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":451,"threads_started":1}
I20260927 18:33:03.605024 8182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:36041
I20260927 18:33:03.605067 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:36041 every 8 connection(s)
I20260927 18:33:03.606339 8182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:03.608440 8317 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34859
I20260927 18:33:03.608673 8317 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:03.609282 8317 heartbeater.cc:511] Master 127.0.97.126:34859 requested a full tablet report, sending...
I20260927 18:33:03.610659 7971 ts_manager.cc:194] Registered new tserver with Master: 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:03.611156 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8182
I20260927 18:33:03.611340 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:33:03.611816 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:38577
I20260927 18:33:03.611879 389 external_mini_cluster.cc:1428] Setting key 394cab80114b1ab46bfc626806e3756b
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260927 18:33:03.658850 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34859
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:04.001348 8321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:04.001717 8321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:04.002015 8321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:04.009950 8321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:04.010069 8321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:04.010161 8321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:33:04.010250 8321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:04.020291 8321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:34859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:04.021621 8321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:04.023151 8321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:04.038095 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:04.038022 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:04.038023 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:04.038740 8321 server_base.cc:1034] running on GCE node
I20260927 18:33:04.039347 8321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:04.040251 8321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:04.041495 8321 hybrid_clock.cc:648] HybridClock initialized: now 1790533984041398 us; error 75 us; skew 500 ppm
I20260927 18:33:04.041857 8321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:04.045013 8321 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260927 18:33:04.046537 8321 webserver.cc:460] Webserver started at http://127.0.97.67:35427/ using document root <none> and password file <none>
I20260927 18:33:04.047221 8321 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:04.047345 8321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:04.047665 8321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:04.050657 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "ccb816b9b4e94d22b2d1213a19670129"
format_stamp: "Formatted at 2026-09-27 18:33:04 on dist-test-slave-n3p4"
server_key: "d2d3a931fe75969f4fd554a6bd1f3c30"
server_key_iv: "daf17694ad63e671a0e0eb5824fb02b6"
server_key_version: "encryptionkey@0"
I20260927 18:33:04.051425 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "ccb816b9b4e94d22b2d1213a19670129"
format_stamp: "Formatted at 2026-09-27 18:33:04 on dist-test-slave-n3p4"
server_key: "d2d3a931fe75969f4fd554a6bd1f3c30"
server_key_iv: "daf17694ad63e671a0e0eb5824fb02b6"
server_key_version: "encryptionkey@0"
I20260927 18:33:04.056608 8321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:33:04.060436 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:04.061870 8321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:33:04.062036 8321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "ccb816b9b4e94d22b2d1213a19670129"
format_stamp: "Formatted at 2026-09-27 18:33:04 on dist-test-slave-n3p4"
server_key: "d2d3a931fe75969f4fd554a6bd1f3c30"
server_key_iv: "daf17694ad63e671a0e0eb5824fb02b6"
server_key_version: "encryptionkey@0"
I20260927 18:33:04.062263 8321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:04.086607 8321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:04.090373 8178 heartbeater.cc:503] Master 127.0.97.126:34859 was elected leader, sending a full tablet report...
I20260927 18:33:04.090811 8321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:04.091094 8321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:04.094117 8321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:04.094218 8321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:04.094350 8321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:04.094444 8321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:04.116747 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.095813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35389 (local address 127.0.97.126:34859)
0927 18:33:04.096026 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.096033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.096357 (+ 324us) server_negotiation.cc:407] Connection header received
0927 18:33:04.098437 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.098444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.098549 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.098653 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:04.101002 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.102121 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.103996 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.104388 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.104526 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.109324 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:04.109329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:04.109332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:04.109365 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:04.112719 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.114092 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.114098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.114101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.114266 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.114816 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.114822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.114827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.115244 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:04.115408 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.116166 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.116386 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:33:04.117767 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.095325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.67:35389)
0927 18:33:04.096215 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.096250 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.098131 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.099021 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.099037 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.099784 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:04.100796 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.100834 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.102305 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.102329 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.103853 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.103864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.104612 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.104619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.104828 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.105537 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:04.105559 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:04.109054 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:04.112953 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.112967 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.112999 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.113847 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.114412 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.114418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.114424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.114684 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.115384 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:04.115396 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:04.115755 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.116022 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.116560 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":440,"threads_started":1}
I20260927 18:33:04.139601 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.124036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.97.67:35967)
0927 18:33:04.124920 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.124944 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.125096 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.125431 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.125436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.125982 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:04.126546 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.126567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.127958 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.127961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.129968 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.129980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.131216 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.131221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.131360 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.132045 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:04.132062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:04.132775 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:04.137055 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.137071 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.137076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.137671 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.138066 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.138072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.138075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.138197 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.138682 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:04.138687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:04.138876 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.139039 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.139283 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":448,"threads_started":1}
I20260927 18:33:04.140010 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.123693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35967 (local address 127.0.97.126:34859)
0927 18:33:04.123855 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.123860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.125046 (+ 1186us) server_negotiation.cc:407] Connection header received
0927 18:33:04.125233 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.125238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.125304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.125450 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:04.126748 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.127772 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.130199 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.131005 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.131148 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.132994 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:04.133005 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:04.133012 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:04.133060 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:04.136890 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.137808 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.137817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.137823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.137957 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.138302 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.138308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.138312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.138570 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:04.138718 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.139209 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.139498 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:33:04.152644 8321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40127
I20260927 18:33:04.152650 8455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40127 every 8 connection(s)
I20260927 18:33:04.153893 8321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:04.155910 8456 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34859
I20260927 18:33:04.156116 8456 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:04.156677 8456 heartbeater.cc:511] Master 127.0.97.126:34859 requested a full tablet report, sending...
I20260927 18:33:04.158106 7971 ts_manager.cc:194] Registered new tserver with Master: ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127)
I20260927 18:33:04.158902 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:35389
I20260927 18:33:04.159632 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8321
I20260927 18:33:04.159823 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:33:04.160332 389 external_mini_cluster.cc:1428] Setting key f8f9831bd45fbcb565ff7e8c9735161a
I20260927 18:33:04.163198 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:33:04.211364 8065 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:04.201104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.0.97.126:34859)
0927 18:33:04.201328 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.201334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.201484 (+ 150us) server_negotiation.cc:407] Connection header received
0927 18:33:04.201655 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.201659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.201724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.201878 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:04.203261 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.204059 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.205263 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.205683 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.205852 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.211135 (+ 5283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54800: BlockingRecv error: recv got EOF from 127.0.0.1:54800 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:04.228277 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.215022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.0.97.126:34859)
0927 18:33:04.215247 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.215253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.215450 (+ 197us) server_negotiation.cc:407] Connection header received
0927 18:33:04.215631 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.215636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.215719 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.215817 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:04.216941 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.217704 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.219015 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.219422 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.219625 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.223245 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:04.223254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:04.223259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:04.223304 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:04.226037 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.226627 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.226631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.226633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.226757 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.227080 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.227083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.227085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.227341 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:04.227528 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.227789 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.227952 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:33:04.232543 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:33:04.239259 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:04.262434 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.253972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.0.97.65:36073)
0927 18:33:04.254608 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.254620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.254653 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:33:04.254761 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.254770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.255036 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.256065 (+ 1029us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.256270 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.257902 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.260292 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.261190 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.261336 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.261570 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.261972 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.262138 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":338,"threads_started":1}
I20260927 18:33:04.262504 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.253457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36073 (local address 127.0.0.1:50664)
0927 18:33:04.253971 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.253988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.254100 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.255303 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.255310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.255343 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.255829 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.255843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.258048 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.258056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.260125 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.260137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.261405 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.261421 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.261671 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.261707 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.261915 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.262114 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":54,"thread_start_us":654,"threads_started":1}
I20260927 18:33:04.264843 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.251308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40127 (local address 127.0.0.1:44148)
0927 18:33:04.253339 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.253359 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.253474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.256903 (+ 3429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.256912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.256933 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.257392 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.257401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.259425 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.259431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.262073 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.262089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.263978 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.263984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.264155 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.264184 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.264275 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.264399 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1888,"mutex_wait_us":305,"thread_start_us":1742,"threads_started":1}
I20260927 18:33:04.264849 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.251947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.97.67:40127)
0927 18:33:04.254982 (+ 3035us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.254991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.255024 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:33:04.255116 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.255127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.255386 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.256886 (+ 1500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.257614 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.259266 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.262268 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.263759 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.263932 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.264179 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.264321 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.264482 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2864,"thread_start_us":3623,"threads_started":1}
I20260927 18:33:04.264953 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.251011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36041 (local address 127.0.0.1:54698)
0927 18:33:04.252385 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.252457 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.252688 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.256626 (+ 3938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.256630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.256655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.257106 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.257123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.258918 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.258925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.261480 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.261497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.264050 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.264057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.264257 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.264295 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.264368 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.264506 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":1,"thread_start_us":830,"threads_started":1}
I20260927 18:33:04.266224 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.251645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.0.97.66:36041)
0927 18:33:04.256031 (+ 4386us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.256040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.256067 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:33:04.256162 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.256171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.256409 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.256617 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.257300 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.258741 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.261677 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.263806 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.265431 (+ 1625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.265635 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.265747 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.265958 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4188,"thread_start_us":4062,"threads_started":1}
I20260927 18:33:04.266669 8108 tablet_service.cc:1467] Processing CreateTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 (DEFAULT_TABLE table=test-table [id=fb908ac698724bffa90adfcb36c3b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.268573 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0f95c739c74e8aa48a19d749a1fca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.268486 8391 tablet_service.cc:1467] Processing CreateTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 (DEFAULT_TABLE table=test-table [id=fb908ac698724bffa90adfcb36c3b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.269177 8247 tablet_service.cc:1467] Processing CreateTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 (DEFAULT_TABLE table=test-table [id=fb908ac698724bffa90adfcb36c3b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.269861 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0f95c739c74e8aa48a19d749a1fca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.270336 8247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0f95c739c74e8aa48a19d749a1fca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.282215 8480 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Bootstrap starting.
I20260927 18:33:04.285533 8480 tablet_bootstrap.cc:654] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.286573 8481 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Bootstrap starting.
I20260927 18:33:04.287079 8480 log.cc:824] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:04.287452 8482 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Bootstrap starting.
I20260927 18:33:04.290386 8480 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: No bootstrap required, opened a new log
I20260927 18:33:04.290689 8480 ts_tablet_manager.cc:1397] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 18:33:04.291355 8482 tablet_bootstrap.cc:654] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.291355 8481 tablet_bootstrap.cc:654] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.292718 8481 log.cc:824] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:04.292716 8482 log.cc:824] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:04.296000 8481 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: No bootstrap required, opened a new log
I20260927 18:33:04.295998 8482 tablet_bootstrap.cc:492] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: No bootstrap required, opened a new log
I20260927 18:33:04.296276 8481 ts_tablet_manager.cc:1397] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 18:33:04.296276 8482 ts_tablet_manager.cc:1397] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 18:33:04.298503 8480 raft_consensus.cc:348] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.298846 8480 raft_consensus.cc:374] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.298966 8480 raft_consensus.cc:729] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.299553 8480 consensus_queue.cc:260] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.307546 8482 raft_consensus.cc:348] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.307512 8481 raft_consensus.cc:348] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.307987 8481 raft_consensus.cc:374] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.307987 8482 raft_consensus.cc:374] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.308102 8482 raft_consensus.cc:729] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.308178 8481 raft_consensus.cc:729] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45de60c2741043f9848d383bbe637ba3, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.308768 8317 heartbeater.cc:503] Master 127.0.97.126:34859 was elected leader, sending a full tablet report...
I20260927 18:33:04.308930 8481 consensus_queue.cc:260] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.309471 8480 ts_tablet_manager.cc:1428] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Time spent starting tablet: real 0.019s user 0.004s sys 0.014s
I20260927 18:33:04.309355 8482 consensus_queue.cc:260] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.311364 8456 heartbeater.cc:503] Master 127.0.97.126:34859 was elected leader, sending a full tablet report...
I20260927 18:33:04.312004 8482 ts_tablet_manager.cc:1428] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260927 18:33:04.313716 8481 ts_tablet_manager.cc:1428] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260927 18:33:04.333253 8179 tablet.cc:2369] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:33:04.349880 8488 raft_consensus.cc:482] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:04.350234 8488 raft_consensus.cc:504] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.351995 8488 leader_election.cc:288] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041), 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073)
W20260927 18:33:04.358573 8318 tablet.cc:2369] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:33:04.358685 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.352274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46343 (local address 127.0.97.66:36041)
0927 18:33:04.352496 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.352502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.353055 (+ 553us) server_negotiation.cc:407] Connection header received
0927 18:33:04.353181 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.353187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.353266 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.353354 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.353922 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.354814 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.356725 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.357647 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.357746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.357897 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.358213 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.358356 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:33:04.358711 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.352389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36041 (local address 127.0.97.67:46343)
0927 18:33:04.352703 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.352719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.352882 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.353409 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.353412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.353460 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.353714 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.353726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.354979 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.354986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.356568 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.356584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.357833 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.357840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.358042 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.358083 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.358195 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.358346 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1}
I20260927 18:33:04.359658 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.352447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33739 (local address 127.0.97.65:36073)
0927 18:33:04.352745 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.352752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.353360 (+ 608us) server_negotiation.cc:407] Connection header received
0927 18:33:04.353515 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.353519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.353616 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.353725 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.354337 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.355251 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.357185 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.358240 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.358364 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.358467 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.359330 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.359477 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260927 18:33:04.359838 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.352425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36073 (local address 127.0.97.67:33739)
0927 18:33:04.353247 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.353262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.353376 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.353736 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.353739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.353756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.354154 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.354167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.355390 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.355393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.357030 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.357043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.358624 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.358630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.359147 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.359172 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.359299 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.359440 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":22,"thread_start_us":451,"threads_started":1}
I20260927 18:33:04.359826 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0f95c739c74e8aa48a19d749a1fca1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a" is_pre_election: true
I20260927 18:33:04.360441 8268 raft_consensus.cc:2454] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:04.360847 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0f95c739c74e8aa48a19d749a1fca1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3" is_pre_election: true
I20260927 18:33:04.361229 8339 leader_election.cc:302] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413d0204bde24189839d3eb9ae6cae3a, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:04.361445 8133 raft_consensus.cc:2454] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:04.361672 8488 raft_consensus.cc:2781] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:04.361850 8488 raft_consensus.cc:482] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:04.361975 8488 raft_consensus.cc:3037] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.364215 8488 raft_consensus.cc:504] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.364755 8488 leader_election.cc:288] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Requested vote from peers 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041), 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073)
I20260927 18:33:04.365206 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0f95c739c74e8aa48a19d749a1fca1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a"
I20260927 18:33:04.365226 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0f95c739c74e8aa48a19d749a1fca1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3"
I20260927 18:33:04.365413 8133 raft_consensus.cc:3037] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.365402 8268 raft_consensus.cc:3037] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.367740 8133 raft_consensus.cc:2454] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:04.367764 8268 raft_consensus.cc:2454] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:04.368109 8337 leader_election.cc:302] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45de60c2741043f9848d383bbe637ba3, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:04.368430 8488 raft_consensus.cc:2781] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:04.368854 8488 raft_consensus.cc:686] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Becoming Leader. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Running, Role: LEADER
I20260927 18:33:04.369328 8488 consensus_queue.cc:237] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } }
I20260927 18:33:04.373925 7969 catalog_manager.cc:5495] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } health_report { overall_health: UNKNOWN } } }
W20260927 18:33:04.405493 8457 tablet.cc:2369] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:33:04.416163 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.410841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.0.97.67:40127)
0927 18:33:04.411070 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.411075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.411093 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:33:04.411321 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.411326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.411410 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.411577 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:04.411922 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.412753 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.414136 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.414541 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.414634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.414985 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.415550 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.415665 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.415789 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.415852 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:33:04.424144 8133 raft_consensus.cc:1261] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.424177 8268 raft_consensus.cc:1261] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.425061 8490 consensus_queue.cc:1035] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.425462 8488 consensus_queue.cc:1035] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.456089 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:04.456418 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:04.460788 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Sending DeleteTablet for 3 replicas of tablet ce0f95c739c74e8aa48a19d749a1fca1
I20260927 18:33:04.461969 8247 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54698
I20260927 18:33:04.462368 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260927 18:33:04.462767 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0f95c739c74e8aa48a19d749a1fca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260927 18:33:04.463887 7969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:33:04.464669 7969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:04.474707 8246 tablet_service.cc:1467] Processing CreateTablet for tablet a4edef97c3bf42a7901e22407ebea8ab (DEFAULT_TABLE table=test-table [id=a137d53f2d6640e6a09984e3e2320f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.474752 8107 tablet_service.cc:1467] Processing CreateTablet for tablet a4edef97c3bf42a7901e22407ebea8ab (DEFAULT_TABLE table=test-table [id=a137d53f2d6640e6a09984e3e2320f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.475062 8385 tablet_service.cc:1467] Processing CreateTablet for tablet a4edef97c3bf42a7901e22407ebea8ab (DEFAULT_TABLE table=test-table [id=a137d53f2d6640e6a09984e3e2320f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:04.475345 8107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4edef97c3bf42a7901e22407ebea8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.475349 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4edef97c3bf42a7901e22407ebea8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.475710 8385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4edef97c3bf42a7901e22407ebea8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.482088 8481 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Bootstrap starting.
I20260927 18:33:04.483793 8481 tablet_bootstrap.cc:654] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.490895 8482 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Bootstrap starting.
I20260927 18:33:04.492797 8482 tablet_bootstrap.cc:654] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.494211 8505 tablet_replica.cc:330] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: stopping tablet replica
I20260927 18:33:04.494793 8505 raft_consensus.cc:2229] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:04.495429 8505 raft_consensus.cc:2258] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.499990 8504 tablet_replica.cc:330] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: stopping tablet replica
I20260927 18:33:04.500509 8504 raft_consensus.cc:2229] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:33:04.500710 8480 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Bootstrap starting.
I20260927 18:33:04.502646 8480 tablet_bootstrap.cc:654] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.504357 8505 ts_tablet_manager.cc:1905] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:04.501266 8504 raft_consensus.cc:2258] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.511211 8504 ts_tablet_manager.cc:1905] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:04.521386 8505 ts_tablet_manager.cc:1918] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:04.521588 8505 log.cc:1196] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/ce0f95c739c74e8aa48a19d749a1fca1
I20260927 18:33:04.522245 8505 ts_tablet_manager.cc:1939] T ce0f95c739c74e8aa48a19d749a1fca1 P 45de60c2741043f9848d383bbe637ba3: Deleting consensus metadata
I20260927 18:33:04.523029 8482 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: No bootstrap required, opened a new log
I20260927 18:33:04.523205 8482 ts_tablet_manager.cc:1397] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.005s
I20260927 18:33:04.525221 8480 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: No bootstrap required, opened a new log
I20260927 18:33:04.525404 8480 ts_tablet_manager.cc:1397] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.007s
I20260927 18:33:04.526751 8480 raft_consensus.cc:348] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.527005 8480 raft_consensus.cc:374] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.527119 8480 raft_consensus.cc:729] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.527595 8482 raft_consensus.cc:348] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.527869 8482 raft_consensus.cc:374] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.527992 8482 raft_consensus.cc:729] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.528323 8480 consensus_queue.cc:260] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.528823 8482 consensus_queue.cc:260] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.529508 8482 ts_tablet_manager.cc:1428] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260927 18:33:04.531631 8480 ts_tablet_manager.cc:1428] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 18:33:04.533084 8481 tablet_bootstrap.cc:492] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: No bootstrap required, opened a new log
I20260927 18:33:04.533263 8481 ts_tablet_manager.cc:1397] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.002s
I20260927 18:33:04.534153 8481 raft_consensus.cc:348] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.534408 8481 raft_consensus.cc:374] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.534510 8481 raft_consensus.cc:729] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45de60c2741043f9848d383bbe637ba3, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.534847 8481 consensus_queue.cc:260] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.538763 8481 ts_tablet_manager.cc:1428] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:33:04.540560 8504 ts_tablet_manager.cc:1918] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:04.540782 8504 log.cc:1196] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/ce0f95c739c74e8aa48a19d749a1fca1
I20260927 18:33:04.541363 8504 ts_tablet_manager.cc:1939] T ce0f95c739c74e8aa48a19d749a1fca1 P ccb816b9b4e94d22b2d1213a19670129: Deleting consensus metadata
I20260927 18:33:04.541414 8503 tablet_replica.cc:330] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: stopping tablet replica
I20260927 18:33:04.541918 8503 raft_consensus.cc:2229] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:04.542235 7955 catalog_manager.cc:4841] TS 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073): tablet ce0f95c739c74e8aa48a19d749a1fca1 (table test-table [id=fb908ac698724bffa90adfcb36c3b471]) successfully deleted
I20260927 18:33:04.542616 8503 raft_consensus.cc:2258] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.543180 7955 catalog_manager.cc:4841] TS ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127): tablet ce0f95c739c74e8aa48a19d749a1fca1 (table test-table [id=fb908ac698724bffa90adfcb36c3b471]) successfully deleted
I20260927 18:33:04.544775 8503 ts_tablet_manager.cc:1905] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:04.551718 8503 ts_tablet_manager.cc:1918] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:04.551888 8503 log.cc:1196] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/ce0f95c739c74e8aa48a19d749a1fca1
I20260927 18:33:04.552454 8503 ts_tablet_manager.cc:1939] T ce0f95c739c74e8aa48a19d749a1fca1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting consensus metadata
I20260927 18:33:04.554169 7957 catalog_manager.cc:4841] TS 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041): tablet ce0f95c739c74e8aa48a19d749a1fca1 (table test-table [id=fb908ac698724bffa90adfcb36c3b471]) successfully deleted
I20260927 18:33:04.660799 8486 raft_consensus.cc:482] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:04.661058 8486 raft_consensus.cc:504] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.663112 8486 leader_election.cc:288] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127)
I20260927 18:33:04.671482 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.664552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39287 (local address 127.0.97.67:40127)
0927 18:33:04.664732 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.664739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.665368 (+ 629us) server_negotiation.cc:407] Connection header received
0927 18:33:04.665629 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.665637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.665716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.665829 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.666511 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.667376 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.669164 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.670355 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.670491 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.670588 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.671011 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.671156 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:33:04.671586 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.663532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46987 (local address 127.0.97.65:36073)
0927 18:33:04.663774 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.663783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.664347 (+ 564us) server_negotiation.cc:407] Connection header received
0927 18:33:04.664666 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.664675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.664779 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.664857 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:04.665693 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.666702 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.669486 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.670687 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.670824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.670926 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.671204 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.671323 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:33:04.671686 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.663430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36073 (local address 127.0.97.66:46987)
0927 18:33:04.664252 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.664274 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.664451 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.665083 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.665090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.665118 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.665473 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.665483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.666850 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.666857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.669322 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.669337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.670898 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.670904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.671056 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.671078 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.671183 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.671305 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":26,"thread_start_us":518,"threads_started":1}
I20260927 18:33:04.672269 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.664407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40127 (local address 127.0.97.66:39287)
0927 18:33:04.665233 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.665254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.665467 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.665874 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.665879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.665906 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:04.666334 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.666347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.667517 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.667524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.669009 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.669020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.670570 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.670577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.670767 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.670806 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.671727 (+ 921us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.671870 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":24,"thread_start_us":474,"threads_started":1}
I20260927 18:33:04.672488 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4edef97c3bf42a7901e22407ebea8ab" candidate_uuid: "413d0204bde24189839d3eb9ae6cae3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3" is_pre_election: true
I20260927 18:33:04.672756 8133 raft_consensus.cc:2454] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413d0204bde24189839d3eb9ae6cae3a in term 0.
I20260927 18:33:04.673110 8411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4edef97c3bf42a7901e22407ebea8ab" candidate_uuid: "413d0204bde24189839d3eb9ae6cae3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb816b9b4e94d22b2d1213a19670129" is_pre_election: true
I20260927 18:33:04.673296 8198 leader_election.cc:302] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413d0204bde24189839d3eb9ae6cae3a, 45de60c2741043f9848d383bbe637ba3; no voters:
I20260927 18:33:04.673491 8411 raft_consensus.cc:2454] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413d0204bde24189839d3eb9ae6cae3a in term 0.
I20260927 18:33:04.673705 8486 raft_consensus.cc:2781] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:04.673851 8486 raft_consensus.cc:482] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:04.673975 8486 raft_consensus.cc:3037] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.676108 8486 raft_consensus.cc:504] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.676676 8486 leader_election.cc:288] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [CANDIDATE]: Term 1 election: Requested vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127)
I20260927 18:33:04.677138 8411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4edef97c3bf42a7901e22407ebea8ab" candidate_uuid: "413d0204bde24189839d3eb9ae6cae3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb816b9b4e94d22b2d1213a19670129"
I20260927 18:33:04.677350 8411 raft_consensus.cc:3037] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.677343 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4edef97c3bf42a7901e22407ebea8ab" candidate_uuid: "413d0204bde24189839d3eb9ae6cae3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3"
I20260927 18:33:04.677534 8133 raft_consensus.cc:3037] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.680434 8133 raft_consensus.cc:2454] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413d0204bde24189839d3eb9ae6cae3a in term 1.
I20260927 18:33:04.680444 8411 raft_consensus.cc:2454] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413d0204bde24189839d3eb9ae6cae3a in term 1.
I20260927 18:33:04.680778 8198 leader_election.cc:302] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413d0204bde24189839d3eb9ae6cae3a, 45de60c2741043f9848d383bbe637ba3; no voters:
I20260927 18:33:04.681061 8486 raft_consensus.cc:2781] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:04.681512 8486 raft_consensus.cc:686] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 1 LEADER]: Becoming Leader. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Running, Role: LEADER
I20260927 18:33:04.681980 8486 consensus_queue.cc:237] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.685905 7971 catalog_manager.cc:5495] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66). New cstate: current_term: 1 leader_uuid: "413d0204bde24189839d3eb9ae6cae3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:33:04 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:04.774044 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.760200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34859 (local address 127.0.0.1:51996)
0927 18:33:04.760870 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.760898 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.761063 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.761480 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.761484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.762266 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:04.762507 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.762516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.763715 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.763723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.764640 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.764651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.765342 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.765348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.765655 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.766398 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:04.766414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:04.768676 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:04.771519 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.771530 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.771538 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.771910 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.772316 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:04.772324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:04.772327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:04.772460 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:04.772960 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:04.772968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:04.773134 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.773287 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.773560 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":49,"thread_start_us":360,"threads_started":1}
I20260927 18:33:04.774044 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.760275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.97.126:34859)
0927 18:33:04.761241 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.761245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.761261 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:33:04.761300 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.761303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.761351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.761460 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:04.762688 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.763586 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.764776 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.765178 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.765531 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.768848 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:04.768853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:04.768856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:04.768882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:04.771374 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.772036 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.772041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.772045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.772211 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:04.772570 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:04.772575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:04.772578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:04.772835 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:04.772975 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.773373 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.773677 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":840,"thread_start_us":648,"threads_started":1}
I20260927 18:33:04.779904 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:33:04.780895 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:04.787657 8108 tablet_service.cc:1467] Processing CreateTablet for tablet f3841e5de0c141999328f5ce7b0210d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ed6fa2c9f254400b399c2b9ddf8ba26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:33:04.787961 8247 tablet_service.cc:1467] Processing CreateTablet for tablet f3841e5de0c141999328f5ce7b0210d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ed6fa2c9f254400b399c2b9ddf8ba26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:33:04.787971 8391 tablet_service.cc:1467] Processing CreateTablet for tablet f3841e5de0c141999328f5ce7b0210d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ed6fa2c9f254400b399c2b9ddf8ba26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:33:04.788359 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3841e5de0c141999328f5ce7b0210d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.788477 8247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3841e5de0c141999328f5ce7b0210d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.788488 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3841e5de0c141999328f5ce7b0210d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:04.795778 8482 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: Bootstrap starting.
I20260927 18:33:04.795765 8480 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: Bootstrap starting.
I20260927 18:33:04.797463 8482 tablet_bootstrap.cc:654] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.797458 8480 tablet_bootstrap.cc:654] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.799997 8481 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: Bootstrap starting.
I20260927 18:33:04.801951 8481 tablet_bootstrap.cc:654] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:04.807749 8482 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: No bootstrap required, opened a new log
I20260927 18:33:04.807929 8482 ts_tablet_manager.cc:1397] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260927 18:33:04.808573 8482 raft_consensus.cc:348] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.808804 8482 raft_consensus.cc:374] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.808893 8482 raft_consensus.cc:729] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.809158 8482 consensus_queue.cc:260] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.809585 8498 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.810062 8498 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.810364 8482 ts_tablet_manager.cc:1428] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:33:04.811487 8480 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: No bootstrap required, opened a new log
I20260927 18:33:04.811666 8480 ts_tablet_manager.cc:1397] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260927 18:33:04.812479 8480 raft_consensus.cc:348] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.812728 8480 raft_consensus.cc:374] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.812808 8480 raft_consensus.cc:729] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.813066 8480 consensus_queue.cc:260] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.813722 8480 ts_tablet_manager.cc:1428] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:33:04.814275 8486 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.814580 8486 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.816419 8481 tablet_bootstrap.cc:492] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: No bootstrap required, opened a new log
I20260927 18:33:04.816617 8481 ts_tablet_manager.cc:1397] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260927 18:33:04.817476 8481 raft_consensus.cc:348] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.817745 8481 raft_consensus.cc:374] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:04.818131 8481 raft_consensus.cc:729] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45de60c2741043f9848d383bbe637ba3, State: Initialized, Role: FOLLOWER
I20260927 18:33:04.818508 8481 consensus_queue.cc:260] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.819132 8487 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.819465 8487 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.820034 8481 ts_tablet_manager.cc:1428] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:04.822395 8498 raft_consensus.cc:482] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:04.822599 8498 raft_consensus.cc:504] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.823175 8498 leader_election.cc:288] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:04.823616 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3841e5de0c141999328f5ce7b0210d2" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3" is_pre_election: true
I20260927 18:33:04.823843 8133 raft_consensus.cc:2454] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:04.823823 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3841e5de0c141999328f5ce7b0210d2" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a" is_pre_election: true
I20260927 18:33:04.824059 8268 raft_consensus.cc:2454] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:04.824366 8337 leader_election.cc:302] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45de60c2741043f9848d383bbe637ba3, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:04.824656 8498 raft_consensus.cc:2781] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:04.824743 8498 raft_consensus.cc:482] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:04.824811 8498 raft_consensus.cc:3037] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.827212 8498 raft_consensus.cc:504] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.827762 8498 leader_election.cc:288] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Requested vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:04.828078 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3841e5de0c141999328f5ce7b0210d2" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3"
I20260927 18:33:04.828306 8133 raft_consensus.cc:3037] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.828328 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3841e5de0c141999328f5ce7b0210d2" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a"
I20260927 18:33:04.828531 8268 raft_consensus.cc:3037] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:04.831141 8133 raft_consensus.cc:2454] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:04.831478 8268 raft_consensus.cc:2454] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:04.831574 8337 leader_election.cc:302] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45de60c2741043f9848d383bbe637ba3, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:04.831902 8498 raft_consensus.cc:2781] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:04.832441 8498 raft_consensus.cc:686] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Becoming Leader. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Running, Role: LEADER
I20260927 18:33:04.832703 8498 consensus_queue.cc:237] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:04.834197 8490 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: TxnStatusTablet state changed. Reason: New leader ccb816b9b4e94d22b2d1213a19670129. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.834537 8490 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:33:04.836400 7969 catalog_manager.cc:5495] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } health_report { overall_health: HEALTHY } } }
I20260927 18:33:04.839119 8524 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:33:04.839397 8524 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:33:04.860723 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.854446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40127 (local address 127.0.0.1:43606)
0927 18:33:04.854685 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.854708 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.854860 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.855308 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.855313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.855328 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:33:04.855540 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.855555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.856551 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.856559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.858462 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.858478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.859347 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.859353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.859499 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.859549 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.860189 (+ 640us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.860203 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.860300 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.860446 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:33:04.860724 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.854495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.0.97.67:40127)
0927 18:33:04.854734 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.854738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.854760 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:33:04.855084 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.855088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.855166 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.855237 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:04.855699 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.856406 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.858606 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.859178 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.859345 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.859752 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.859854 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.859948 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.860336 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.860411 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 18:33:04.863930 8133 raft_consensus.cc:1261] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.864193 8268 raft_consensus.cc:1261] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.864529 8490 consensus_queue.cc:1035] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.864790 8498 consensus_queue.cc:1035] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.870621 8486 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: TxnStatusTablet state changed. Reason: New leader ccb816b9b4e94d22b2d1213a19670129. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.870872 8486 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.873029 8493 mvcc.cc:205] Tried to move back new op lower bound from 7334027201997185024 to 7334027201876860928. Current Snapshot: MvccSnapshot[applied={T|T < 7334027201997185024}]
I20260927 18:33:04.876132 8490 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.876605 8490 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:33:04.877219 8498 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.877513 8498 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P ccb816b9b4e94d22b2d1213a19670129: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:33:04.877765 8487 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: TxnStatusTablet state changed. Reason: New leader ccb816b9b4e94d22b2d1213a19670129. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.878082 8487 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.879655 8486 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.879946 8486 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 413d0204bde24189839d3eb9ae6cae3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.882889 8487 tablet_replica.cc:439] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } } }
I20260927 18:33:04.883082 8487 tablet_replica.cc:442] T f3841e5de0c141999328f5ce7b0210d2 P 45de60c2741043f9848d383bbe637ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:33:04.889390 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.883764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.97.66:36041)
0927 18:33:04.884296 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.884304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.884328 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:33:04.884398 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.884401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.884471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.884632 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:04.884988 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.885872 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.887142 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.887587 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.887732 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.888194 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.888744 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.888888 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.888968 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.889039 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":457,"threads_started":1}
I20260927 18:33:04.907199 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.901317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39011 (local address 127.0.97.67:40127)
0927 18:33:04.901516 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.901523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.902235 (+ 712us) server_negotiation.cc:407] Connection header received
0927 18:33:04.902464 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.902468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.902530 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.902642 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:04.903155 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.903946 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.905287 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.905718 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.905926 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.906368 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.906511 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.906618 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.906761 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.906839 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:33:04.907233 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.901289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40127 (local address 127.0.97.66:39011)
0927 18:33:04.902131 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.902147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.902310 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.902657 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.902661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.902671 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:33:04.903020 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.903029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.904077 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.904084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.905139 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.905150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.905984 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.905992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.906125 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.906188 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.906624 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.906643 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.906739 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.906865 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":506,"threads_started":1}
I20260927 18:33:04.916767 8133 raft_consensus.cc:1261] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Refusing update from remote peer 413d0204bde24189839d3eb9ae6cae3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.916862 8411 raft_consensus.cc:1261] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Refusing update from remote peer 413d0204bde24189839d3eb9ae6cae3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:04.917456 8511 consensus_queue.cc:1035] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.917863 8486 consensus_queue.cc:1035] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:04.926034 8494 mvcc.cc:205] Tried to move back new op lower bound from 7334027202212286464 to 7334027201261830144. Current Snapshot: MvccSnapshot[applied={T|T < 7334027202212286464}]
I20260927 18:33:04.926183 8495 mvcc.cc:205] Tried to move back new op lower bound from 7334027202212286464 to 7334027201261830144. Current Snapshot: MvccSnapshot[applied={T|T < 7334027202212286464}]
I20260927 18:33:04.954656 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.947705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52375 (local address 127.0.97.66:36041)
0927 18:33:04.947861 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:04.947867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:04.948755 (+ 888us) server_negotiation.cc:407] Connection header received
0927 18:33:04.948908 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:04.948913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:04.949000 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:04.949119 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:04.949771 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.950838 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.952827 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.953242 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.953402 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.953905 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.954031 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.954086 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:04.954289 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:33:04.954367 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:33:04.954777 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:04.947781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36041 (local address 127.0.97.67:52375)
0927 18:33:04.948653 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:04.948671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:04.948786 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:04.949224 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:04.949229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:04.949252 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:33:04.949584 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.949594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.951014 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.951019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.952682 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:04.952696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:04.953449 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:04.953455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:04.953632 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:04.953697 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:33:04.954157 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:33:04.954166 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:33:04.954234 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:33:04.954366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":537,"threads_started":1}
I20260927 18:33:04.990425 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:04.990690 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:04.994112 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Sending DeleteTablet for 3 replicas of tablet a4edef97c3bf42a7901e22407ebea8ab
I20260927 18:33:04.995002 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet a4edef97c3bf42a7901e22407ebea8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260927 18:33:04.995136 8247 tablet_service.cc:1514] Processing DeleteTablet for tablet a4edef97c3bf42a7901e22407ebea8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54698
I20260927 18:33:04.995285 8504 tablet_replica.cc:330] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: stopping tablet replica
I20260927 18:33:04.995383 8503 tablet_replica.cc:330] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: stopping tablet replica
I20260927 18:33:04.995509 8504 raft_consensus.cc:2229] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:04.995568 8503 raft_consensus.cc:2229] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:33:04.995702 8504 raft_consensus.cc:2258] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.995702 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet a4edef97c3bf42a7901e22407ebea8ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:04 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260927 18:33:04.995986 8505 tablet_replica.cc:330] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: stopping tablet replica
I20260927 18:33:04.996078 8503 raft_consensus.cc:2258] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.996179 8505 raft_consensus.cc:2229] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:04.996385 8505 raft_consensus.cc:2258] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:04.997078 8504 ts_tablet_manager.cc:1905] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:04.997200 8503 ts_tablet_manager.cc:1905] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.003577 8503 ts_tablet_manager.cc:1918] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.003720 8503 log.cc:1196] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/a4edef97c3bf42a7901e22407ebea8ab
I20260927 18:33:05.004125 8503 ts_tablet_manager.cc:1939] T a4edef97c3bf42a7901e22407ebea8ab P 413d0204bde24189839d3eb9ae6cae3a: Deleting consensus metadata
I20260927 18:33:05.004930 8504 ts_tablet_manager.cc:1918] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.005115 8504 log.cc:1196] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/a4edef97c3bf42a7901e22407ebea8ab
I20260927 18:33:05.005147 7957 catalog_manager.cc:4841] TS 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041): tablet a4edef97c3bf42a7901e22407ebea8ab (table test-table [id=a137d53f2d6640e6a09984e3e2320f0c]) successfully deleted
I20260927 18:33:05.005537 8504 ts_tablet_manager.cc:1939] T a4edef97c3bf42a7901e22407ebea8ab P ccb816b9b4e94d22b2d1213a19670129: Deleting consensus metadata
I20260927 18:33:05.006699 7955 catalog_manager.cc:4841] TS ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127): tablet a4edef97c3bf42a7901e22407ebea8ab (table test-table [id=a137d53f2d6640e6a09984e3e2320f0c]) successfully deleted
I20260927 18:33:05.007359 8505 ts_tablet_manager.cc:1905] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.015368 8505 ts_tablet_manager.cc:1918] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.015535 8505 log.cc:1196] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/a4edef97c3bf42a7901e22407ebea8ab
I20260927 18:33:05.015950 8505 ts_tablet_manager.cc:1939] T a4edef97c3bf42a7901e22407ebea8ab P 45de60c2741043f9848d383bbe637ba3: Deleting consensus metadata
I20260927 18:33:05.017095 7955 catalog_manager.cc:4841] TS 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073): tablet a4edef97c3bf42a7901e22407ebea8ab (table test-table [id=a137d53f2d6640e6a09984e3e2320f0c]) successfully deleted
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:33:05.054770 8519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:05.044503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.0.97.126:34859)
0927 18:33:05.044716 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.044723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.045085 (+ 362us) server_negotiation.cc:407] Connection header received
0927 18:33:05.045313 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.045318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.045382 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.045490 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:05.046916 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.047758 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.049217 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.049715 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.049916 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.054591 (+ 4675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52000: BlockingRecv error: recv got EOF from 127.0.0.1:52000 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260927 18:33:05.077275 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.063073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.126:34859)
0927 18:33:05.063288 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.063297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.063730 (+ 433us) server_negotiation.cc:407] Connection header received
0927 18:33:05.063919 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.063924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.063995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.064109 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:05.065283 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.066256 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.067742 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.068173 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.068304 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.071977 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:05.071982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:05.071986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:05.072018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:05.074734 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.075357 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.075364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.075367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.075497 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.075873 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.075880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.075883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.076150 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:05.076403 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:05.076694 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 18:33:05.076914 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:33:05.083475 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:33:05.084163 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:05.091471 8247 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca033593ec447182f90df124833dc1 (DEFAULT_TABLE table=test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.091914 8247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ca033593ec447182f90df124833dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.091840 8108 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca033593ec447182f90df124833dc1 (DEFAULT_TABLE table=test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.092262 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ca033593ec447182f90df124833dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.098191 8391 tablet_service.cc:1467] Processing CreateTablet for tablet 30ca033593ec447182f90df124833dc1 (DEFAULT_TABLE table=test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.098381 8480 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Bootstrap starting.
I20260927 18:33:05.098642 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ca033593ec447182f90df124833dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.099941 8480 tablet_bootstrap.cc:654] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.105346 8482 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Bootstrap starting.
I20260927 18:33:05.106319 8481 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Bootstrap starting.
I20260927 18:33:05.107299 8482 tablet_bootstrap.cc:654] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.107822 8481 tablet_bootstrap.cc:654] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.114249 8481 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: No bootstrap required, opened a new log
I20260927 18:33:05.114390 8481 ts_tablet_manager.cc:1397] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:33:05.115190 8481 raft_consensus.cc:348] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.115433 8481 raft_consensus.cc:374] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.115530 8481 raft_consensus.cc:729] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45de60c2741043f9848d383bbe637ba3, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.115777 8481 consensus_queue.cc:260] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.118256 8481 ts_tablet_manager.cc:1428] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:33:05.119905 8480 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: No bootstrap required, opened a new log
I20260927 18:33:05.120204 8480 ts_tablet_manager.cc:1397] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20260927 18:33:05.121440 8482 tablet_bootstrap.cc:492] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: No bootstrap required, opened a new log
I20260927 18:33:05.121594 8482 ts_tablet_manager.cc:1397] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20260927 18:33:05.121047 8480 raft_consensus.cc:348] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.122364 8482 raft_consensus.cc:348] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.122556 8480 raft_consensus.cc:374] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.122629 8482 raft_consensus.cc:374] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.122730 8482 raft_consensus.cc:729] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.122776 8480 raft_consensus.cc:729] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.122988 8482 consensus_queue.cc:260] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.123106 8480 consensus_queue.cc:260] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.124202 8482 ts_tablet_manager.cc:1428] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:05.127245 8480 ts_tablet_manager.cc:1428] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260927 18:33:05.163246 8488 raft_consensus.cc:482] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:05.163452 8488 raft_consensus.cc:504] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.164167 8488 leader_election.cc:288] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:05.165169 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca033593ec447182f90df124833dc1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3" is_pre_election: true
I20260927 18:33:05.165428 8133 raft_consensus.cc:2454] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:05.165915 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca033593ec447182f90df124833dc1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a" is_pre_election: true
I20260927 18:33:05.166190 8268 raft_consensus.cc:2454] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:05.166695 8339 leader_election.cc:302] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413d0204bde24189839d3eb9ae6cae3a, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:05.167343 8488 raft_consensus.cc:2781] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:05.167469 8488 raft_consensus.cc:482] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:05.167593 8488 raft_consensus.cc:3037] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.170782 8488 raft_consensus.cc:504] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.171823 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca033593ec447182f90df124833dc1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3"
I20260927 18:33:05.172034 8133 raft_consensus.cc:3037] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.175218 8133 raft_consensus.cc:2454] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:05.175650 8337 leader_election.cc:302] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45de60c2741043f9848d383bbe637ba3, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:05.175966 8490 raft_consensus.cc:2781] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:05.176106 8490 raft_consensus.cc:686] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Becoming Leader. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Running, Role: LEADER
I20260927 18:33:05.176401 8490 consensus_queue.cc:237] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.179200 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ca033593ec447182f90df124833dc1" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a"
I20260927 18:33:05.179423 8268 raft_consensus.cc:3037] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.182560 8268 raft_consensus.cc:2454] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:05.183187 8488 leader_election.cc:288] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Requested vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:05.192777 7969 catalog_manager.cc:5495] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } health_report { overall_health: HEALTHY } } }
I20260927 18:33:05.223269 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.216658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.97.67:40127)
0927 18:33:05.216858 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.216865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.217004 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:33:05.217231 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.217236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.217327 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.217419 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:05.217868 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.219094 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.221133 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.221856 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.221980 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.222476 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:05.222612 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:05.222708 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:05.222861 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:33:05.222945 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:33:05.226432 8133 raft_consensus.cc:1261] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:05.226460 8268 raft_consensus.cc:1261] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:05.227171 8490 consensus_queue.cc:1035] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:05.227560 8488 consensus_queue.cc:1035] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:05.250633 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:05.250885 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:05.254218 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Sending DeleteTablet for 3 replicas of tablet 30ca033593ec447182f90df124833dc1
I20260927 18:33:05.255029 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca033593ec447182f90df124833dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260927 18:33:05.255089 8247 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca033593ec447182f90df124833dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54698
I20260927 18:33:05.255352 8505 tablet_replica.cc:330] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: stopping tablet replica
I20260927 18:33:05.255402 8503 tablet_replica.cc:330] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: stopping tablet replica
I20260927 18:33:05.255541 8505 raft_consensus.cc:2229] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:05.255609 8503 raft_consensus.cc:2229] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:05.255816 8505 raft_consensus.cc:2258] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.255837 8503 raft_consensus.cc:2258] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.256714 8503 ts_tablet_manager.cc:1905] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.257081 8505 ts_tablet_manager.cc:1905] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.262822 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ca033593ec447182f90df124833dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260927 18:33:05.263159 8504 tablet_replica.cc:330] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: stopping tablet replica
I20260927 18:33:05.263355 8504 raft_consensus.cc:2229] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:33:05.263731 8504 raft_consensus.cc:2258] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.263952 8505 ts_tablet_manager.cc:1918] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:05.264122 8505 log.cc:1196] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/30ca033593ec447182f90df124833dc1
I20260927 18:33:05.263804 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:33:05.264570 8504 ts_tablet_manager.cc:1905] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.264784 8505 ts_tablet_manager.cc:1939] T 30ca033593ec447182f90df124833dc1 P 45de60c2741043f9848d383bbe637ba3: Deleting consensus metadata
W20260927 18:33:05.264765 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:05.265921 7955 catalog_manager.cc:4841] TS 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073): tablet 30ca033593ec447182f90df124833dc1 (table test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]) successfully deleted
I20260927 18:33:05.267334 8503 ts_tablet_manager.cc:1918] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:05.267457 8503 log.cc:1196] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/30ca033593ec447182f90df124833dc1
I20260927 18:33:05.267880 8503 ts_tablet_manager.cc:1939] T 30ca033593ec447182f90df124833dc1 P 413d0204bde24189839d3eb9ae6cae3a: Deleting consensus metadata
I20260927 18:33:05.268952 7957 catalog_manager.cc:4841] TS 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041): tablet 30ca033593ec447182f90df124833dc1 (table test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]) successfully deleted
I20260927 18:33:05.271314 8504 ts_tablet_manager.cc:1918] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:05.271440 8504 log.cc:1196] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/30ca033593ec447182f90df124833dc1
I20260927 18:33:05.271853 8504 ts_tablet_manager.cc:1939] T 30ca033593ec447182f90df124833dc1 P ccb816b9b4e94d22b2d1213a19670129: Deleting consensus metadata
I20260927 18:33:05.272856 7955 catalog_manager.cc:4841] TS ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127): tablet 30ca033593ec447182f90df124833dc1 (table test-table [id=1d7db5d2d03f4568a3a0f81b9aa2511e]) successfully deleted
I20260927 18:33:05.275652 8108 tablet_service.cc:1467] Processing CreateTablet for tablet d041e1aafe46485c874758a5d8b2af3b (DEFAULT_TABLE table=test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.275876 8247 tablet_service.cc:1467] Processing CreateTablet for tablet d041e1aafe46485c874758a5d8b2af3b (DEFAULT_TABLE table=test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.276094 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d041e1aafe46485c874758a5d8b2af3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.276281 8247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d041e1aafe46485c874758a5d8b2af3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.282539 8391 tablet_service.cc:1467] Processing CreateTablet for tablet d041e1aafe46485c874758a5d8b2af3b (DEFAULT_TABLE table=test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:05.283011 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d041e1aafe46485c874758a5d8b2af3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:05.283320 8480 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Bootstrap starting.
I20260927 18:33:05.285033 8480 tablet_bootstrap.cc:654] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.290673 8481 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Bootstrap starting.
I20260927 18:33:05.292320 8481 tablet_bootstrap.cc:654] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.295883 8480 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: No bootstrap required, opened a new log
I20260927 18:33:05.295889 8482 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Bootstrap starting.
I20260927 18:33:05.296145 8480 ts_tablet_manager.cc:1397] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260927 18:33:05.296988 8480 raft_consensus.cc:348] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.297334 8480 raft_consensus.cc:374] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.297470 8480 raft_consensus.cc:729] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413d0204bde24189839d3eb9ae6cae3a, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.297590 8482 tablet_bootstrap.cc:654] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:05.297883 8480 consensus_queue.cc:260] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.299479 8480 ts_tablet_manager.cc:1428] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:05.307593 8482 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: No bootstrap required, opened a new log
I20260927 18:33:05.307770 8482 ts_tablet_manager.cc:1397] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260927 18:33:05.308596 8482 raft_consensus.cc:348] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.308842 8482 raft_consensus.cc:374] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.308921 8482 raft_consensus.cc:729] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.309154 8482 consensus_queue.cc:260] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.310864 8481 tablet_bootstrap.cc:492] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: No bootstrap required, opened a new log
I20260927 18:33:05.311087 8481 ts_tablet_manager.cc:1397] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.005s
I20260927 18:33:05.311883 8481 raft_consensus.cc:348] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.312206 8481 raft_consensus.cc:374] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:05.312323 8481 raft_consensus.cc:729] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45de60c2741043f9848d383bbe637ba3, State: Initialized, Role: FOLLOWER
I20260927 18:33:05.312659 8482 ts_tablet_manager.cc:1428] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 18:33:05.312611 8481 consensus_queue.cc:260] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.313335 8481 ts_tablet_manager.cc:1428] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 18:33:05.660254 8490 raft_consensus.cc:482] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:05.660439 8490 raft_consensus.cc:504] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.661114 8490 leader_election.cc:288] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:05.661684 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d041e1aafe46485c874758a5d8b2af3b" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3" is_pre_election: true
I20260927 18:33:05.661748 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d041e1aafe46485c874758a5d8b2af3b" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a" is_pre_election: true
I20260927 18:33:05.661962 8133 raft_consensus.cc:2454] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:05.662030 8268 raft_consensus.cc:2454] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 0.
I20260927 18:33:05.662537 8337 leader_election.cc:302] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45de60c2741043f9848d383bbe637ba3, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:05.662835 8490 raft_consensus.cc:2781] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:05.662985 8490 raft_consensus.cc:482] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:05.663111 8490 raft_consensus.cc:3037] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.665297 8490 raft_consensus.cc:504] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.666035 8490 leader_election.cc:288] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Requested vote from peers 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073), 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041)
I20260927 18:33:05.666577 8133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d041e1aafe46485c874758a5d8b2af3b" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45de60c2741043f9848d383bbe637ba3"
I20260927 18:33:05.666581 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d041e1aafe46485c874758a5d8b2af3b" candidate_uuid: "ccb816b9b4e94d22b2d1213a19670129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413d0204bde24189839d3eb9ae6cae3a"
I20260927 18:33:05.666798 8133 raft_consensus.cc:3037] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.666800 8268 raft_consensus.cc:3037] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:05.668807 8268 raft_consensus.cc:2454] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:05.668990 8133 raft_consensus.cc:2454] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb816b9b4e94d22b2d1213a19670129 in term 1.
I20260927 18:33:05.669276 8339 leader_election.cc:302] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413d0204bde24189839d3eb9ae6cae3a, ccb816b9b4e94d22b2d1213a19670129; no voters:
I20260927 18:33:05.669636 8490 raft_consensus.cc:2781] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:05.669817 8490 raft_consensus.cc:686] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Becoming Leader. State: Replica: ccb816b9b4e94d22b2d1213a19670129, State: Running, Role: LEADER
I20260927 18:33:05.670111 8490 consensus_queue.cc:237] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } }
I20260927 18:33:05.672816 7969 catalog_manager.cc:5495] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "ccb816b9b4e94d22b2d1213a19670129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb816b9b4e94d22b2d1213a19670129" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40127 } health_report { overall_health: HEALTHY } } }
I20260927 18:33:05.699390 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.694591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52209 (local address 127.0.97.67:40127)
0927 18:33:05.694744 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.694748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.695262 (+ 514us) server_negotiation.cc:407] Connection header received
0927 18:33:05.695381 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.695384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.695506 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.695632 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:05.695995 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.696693 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.698016 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.698406 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.698502 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.698813 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:05.698927 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:05.698990 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:05.699108 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:33:05.699162 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:33:05.699442 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.694558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40127 (local address 127.0.97.67:52209)
0927 18:33:05.695182 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:05.695198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:05.695301 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:05.695651 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:05.695654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:05.695663 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:33:05.695866 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:05.695873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.696806 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.696809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:05.697878 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:05.697889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.698550 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.698554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:05.698670 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.698701 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:33:05.699009 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:33:05.699017 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:33:05.699079 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:33:05.699168 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":351,"threads_started":1}
I20260927 18:33:05.707414 8268 raft_consensus.cc:1261] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:05.707475 8133 raft_consensus.cc:1261] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Refusing update from remote peer ccb816b9b4e94d22b2d1213a19670129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:05.707947 8564 consensus_queue.cc:1035] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413d0204bde24189839d3eb9ae6cae3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:05.708274 8488 consensus_queue.cc:1035] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45de60c2741043f9848d383bbe637ba3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:05.750718 7969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:05.750945 7969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:05.753651 7969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e13b858534deb998de4ea68271f71: Sending DeleteTablet for 3 replicas of tablet d041e1aafe46485c874758a5d8b2af3b
I20260927 18:33:05.754408 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet d041e1aafe46485c874758a5d8b2af3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260927 18:33:05.754617 8247 tablet_service.cc:1514] Processing DeleteTablet for tablet d041e1aafe46485c874758a5d8b2af3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54698
I20260927 18:33:05.754634 8385 tablet_service.cc:1514] Processing DeleteTablet for tablet d041e1aafe46485c874758a5d8b2af3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:05 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260927 18:33:05.754688 8505 tablet_replica.cc:330] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: stopping tablet replica
I20260927 18:33:05.754889 8504 tablet_replica.cc:330] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: stopping tablet replica
I20260927 18:33:05.754895 8503 tablet_replica.cc:330] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: stopping tablet replica
I20260927 18:33:05.754915 8505 raft_consensus.cc:2229] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:05.755089 8503 raft_consensus.cc:2229] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:05.755090 8504 raft_consensus.cc:2229] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:33:05.755126 8505 raft_consensus.cc:2258] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.755264 8503 raft_consensus.cc:2258] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.755384 8504 raft_consensus.cc:2258] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:05.756318 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8042
I20260927 18:33:05.756338 8504 ts_tablet_manager.cc:1905] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.756476 8503 ts_tablet_manager.cc:1905] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.756471 8505 ts_tablet_manager.cc:1905] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:05.763284 8503 ts_tablet_manager.cc:1918] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.763464 8503 log.cc:1196] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/d041e1aafe46485c874758a5d8b2af3b
I20260927 18:33:05.763890 8503 ts_tablet_manager.cc:1939] T d041e1aafe46485c874758a5d8b2af3b P 413d0204bde24189839d3eb9ae6cae3a: Deleting consensus metadata
I20260927 18:33:05.763921 8504 ts_tablet_manager.cc:1918] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.764092 8504 log.cc:1196] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/d041e1aafe46485c874758a5d8b2af3b
I20260927 18:33:05.764541 8504 ts_tablet_manager.cc:1939] T d041e1aafe46485c874758a5d8b2af3b P ccb816b9b4e94d22b2d1213a19670129: Deleting consensus metadata
I20260927 18:33:05.764640 8505 ts_tablet_manager.cc:1918] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:33:05.765429 8505 log.cc:1196] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/d041e1aafe46485c874758a5d8b2af3b
I20260927 18:33:05.765396 7957 catalog_manager.cc:4841] TS 413d0204bde24189839d3eb9ae6cae3a (127.0.97.66:36041): tablet d041e1aafe46485c874758a5d8b2af3b (table test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]) successfully deleted
I20260927 18:33:05.765699 7955 catalog_manager.cc:4841] TS ccb816b9b4e94d22b2d1213a19670129 (127.0.97.67:40127): tablet d041e1aafe46485c874758a5d8b2af3b (table test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]) successfully deleted
I20260927 18:33:05.766011 8505 ts_tablet_manager.cc:1939] T d041e1aafe46485c874758a5d8b2af3b P 45de60c2741043f9848d383bbe637ba3: Deleting consensus metadata
I20260927 18:33:05.767554 7955 catalog_manager.cc:4841] TS 45de60c2741043f9848d383bbe637ba3 (127.0.97.65:36073): tablet d041e1aafe46485c874758a5d8b2af3b (table test-table [id=9f96faabd6cf4ce0bea4b87192b424fe]) successfully deleted
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260927 18:33:05.786612 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.758967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.0.97.65:36073)
0927 18:33:05.770146 (+ 11179us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.770150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.770165 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:33:05.770207 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.770210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.770273 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.770358 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:05.771511 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.772259 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.773412 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.773890 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.773989 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.781134 (+ 7145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:05.781149 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:05.781165 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:05.781197 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:05.784244 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.784830 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.784837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.784839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.784993 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.785322 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.785326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.785328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.785661 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:05.785945 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:05.786191 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:33:05.786349 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11083,"thread_start_us":10984,"threads_started":1}
I20260927 18:33:05.787256 8173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:05.884749 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8042
I20260927 18:33:05.903203 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8182
Sep 27 18:33:05 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
I20260927 18:33:05.918818 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:05.904051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.97.66:36041)
0927 18:33:05.904637 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:05.904642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:05.904665 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:33:05.904723 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:05.904727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:05.904802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:05.904911 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:05.906184 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.906855 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.908212 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:05.908693 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:05.908803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:05.913036 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:05.913051 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:05.913067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:05.913106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:05.915987 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.916603 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.916611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.916613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.916752 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:05.917185 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:05.917189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:05.917192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:05.917623 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:05.918042 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:05.918292 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:33:05.918545 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":269,"threads_started":1}
I20260927 18:33:05.919471 8312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:06.042117 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8182
I20260927 18:33:06.064841 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8321
Sep 27 18:33:06 dist-test-slave-n3p4 krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
I20260927 18:33:06.080265 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:06.065872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.0.97.67:40127)
0927 18:33:06.066020 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:06.066024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:06.066042 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:33:06.066161 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:06.066164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:06.066225 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:06.066317 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:06.067715 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:06.068389 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:06.069679 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:06.070542 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:06.070645 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:06.074810 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:06.074824 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:06.074840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:06.074875 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:06.077591 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:06.078315 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:06.078323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:06.078325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:06.078476 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:06.078858 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:06.078862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:06.078865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:06.079244 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:06.079530 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:06.079739 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:33:06.079921 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:33:06.080909 8451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:06.225404 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8321
I20260927 18:33:06.250442 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 7939
I20260927 18:33:06.251714 8011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:06.367990 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 7939
2026-09-27T18:33:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5665 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:33:06 dist-test-slave-n3p4 krb5kdc[8578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:33:06 dist-test-slave-n3p4 krb5kdc[8578](info): set up 2 sockets
Sep 27 18:33:06 dist-test-slave-n3p4 krb5kdc[8578](info): commencing operation
krb5kdc: starting...
W20260927 18:33:07.470875 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:33:07 dist-test-slave-n3p4 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:33:07.680953 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36783
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:08.011442 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:08.011754 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:08.011977 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:08.019346 8591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:08.019520 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:08.019611 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:08.019671 8591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:08.019727 8591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:08.029484 8591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:08.030885 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:08.032358 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:08.044227 8596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:08.044294 8599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:08.044380 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:08.045482 8591 server_base.cc:1034] running on GCE node
I20260927 18:33:08.046332 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:08.047755 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:08.048977 8591 hybrid_clock.cc:648] HybridClock initialized: now 1790533988048926 us; error 40 us; skew 500 ppm
I20260927 18:33:08.049357 8591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:33:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1748 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:33:08 dist-test-slave-n3p4 krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:33:08 dist-test-slave-n3p4 krb5kdc[8605](info): set up 2 sockets
Sep 27 18:33:08 dist-test-slave-n3p4 krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260927 18:33:09.222875 389 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:33:09 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:33:09.448760 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42537
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:09.767793 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:09.768085 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:09.768282 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:09.775435 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:09.775544 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:09.775655 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:09.775764 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:09.775840 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:09.785509 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:09.786885 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:09.788154 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:09.800037 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:09.800065 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:09.800225 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:09.800966 8618 server_base.cc:1034] running on GCE node
I20260927 18:33:09.801693 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:09.803117 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:09.804325 8618 hybrid_clock.cc:648] HybridClock initialized: now 1790533989804277 us; error 41 us; skew 500 ppm
I20260927 18:33:09.804661 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:09 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:09.807852 8618 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:33:09.809235 8618 webserver.cc:460] Webserver started at http://127.0.97.126:39021/ using document root <none> and password file <none>
I20260927 18:33:09.809909 8618 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:09.810017 8618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:09.810261 8618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:09.812991 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "d2aeaf1a1351439792a2d2dab191ba99"
format_stamp: "Formatted at 2026-09-27 18:33:09 on dist-test-slave-n3p4"
server_key: "2b29ce2a53c48c34ed69dddd7f7d9a15"
server_key_iv: "e92f9bd2cf7feeac238e0f3c48410b09"
server_key_version: "encryptionkey@0"
I20260927 18:33:09.813689 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "d2aeaf1a1351439792a2d2dab191ba99"
format_stamp: "Formatted at 2026-09-27 18:33:09 on dist-test-slave-n3p4"
server_key: "2b29ce2a53c48c34ed69dddd7f7d9a15"
server_key_iv: "e92f9bd2cf7feeac238e0f3c48410b09"
server_key_version: "encryptionkey@0"
I20260927 18:33:09.818573 8618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:33:09.821662 8633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:09.822979 8618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:33:09.823253 8618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "d2aeaf1a1351439792a2d2dab191ba99"
format_stamp: "Formatted at 2026-09-27 18:33:09 on dist-test-slave-n3p4"
server_key: "2b29ce2a53c48c34ed69dddd7f7d9a15"
server_key_iv: "e92f9bd2cf7feeac238e0f3c48410b09"
server_key_version: "encryptionkey@0"
I20260927 18:33:09.823438 8618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:09.847692 8618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:09.852108 8618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:09.852385 8618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:09.872402 8618 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42537
I20260927 18:33:09.872488 8684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42537 every 8 connection(s)
I20260927 18:33:09.873744 8618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:09.877363 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:09.879051 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8618
I20260927 18:33:09.879274 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:33:09.879889 389 external_mini_cluster.cc:1428] Setting key 0103e40079eea61ec743f7f75557b03f
I20260927 18:33:09.891376 8685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Bootstrap starting.
Sep 27 18:33:09 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:09.894484 8685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:09.895710 8685 log.cc:824] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:09.899025 8685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: No bootstrap required, opened a new log
I20260927 18:33:09.904191 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:09.881916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.0.97.126:42537)
0927 18:33:09.882713 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:09.882732 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:09.882785 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:33:09.884437 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:09.884463 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:09.884949 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:09.885354 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:09.886829 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:09.888862 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:09.890131 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:09.891126 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:09.891297 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:09.895511 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:09.895520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:09.895533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:09.895570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:09.899496 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:09.900242 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:09.900250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:09.900257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:09.900476 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:09.900876 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:09.900881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:09.900885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:09.901585 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:09.901920 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:09.902361 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 18:33:09.902817 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":295,"threads_started":1}
I20260927 18:33:09.908583 8685 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:09.909013 8685 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:09.909112 8685 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2aeaf1a1351439792a2d2dab191ba99, State: Initialized, Role: FOLLOWER
I20260927 18:33:09.909655 8685 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:09.909977 8685 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:09.910111 8685 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:09.910270 8685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:09.912362 8685 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:09.912868 8685 leader_election.cc:302] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2aeaf1a1351439792a2d2dab191ba99; no voters:
I20260927 18:33:09.913416 8685 leader_election.cc:288] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:33:09.913579 8690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:09.914273 8690 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 LEADER]: Becoming Leader. State: Replica: d2aeaf1a1351439792a2d2dab191ba99, State: Running, Role: LEADER
I20260927 18:33:09.914908 8690 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:09.915405 8685 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:09.918061 8691 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2aeaf1a1351439792a2d2dab191ba99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } } }
I20260927 18:33:09.918411 8691 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:09.918139 8692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2aeaf1a1351439792a2d2dab191ba99. Latest consensus state: current_term: 1 leader_uuid: "d2aeaf1a1351439792a2d2dab191ba99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } } }
I20260927 18:33:09.918533 8692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:09.919287 8697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:09.924872 8697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:09.933212 8697 catalog_manager.cc:1347] Generated new cluster ID: 763366c6325a4619bd9cf73ed9613963
I20260927 18:33:09.933318 8697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:09.948741 8697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:33:09.950069 8697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:09.955116 8697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Generated new TSK 0
I20260927 18:33:09.955838 8697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:33:10.013617 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:10.347327 8711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:10.347673 8711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:10.347859 8711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:10.355324 8711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:10.355505 8711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:10.355687 8711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:10.365553 8711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:10.366919 8711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:10.368263 8711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:10.382083 8717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:10.382062 8716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:10.382145 8719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:10.382896 8711 server_base.cc:1034] running on GCE node
I20260927 18:33:10.383407 8711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:10.384269 8711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:10.385456 8711 hybrid_clock.cc:648] HybridClock initialized: now 1790533990385401 us; error 49 us; skew 500 ppm
I20260927 18:33:10.385761 8711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:10 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:10.389231 8711 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:33:10.390669 8711 webserver.cc:460] Webserver started at http://127.0.97.65:43769/ using document root <none> and password file <none>
I20260927 18:33:10.391324 8711 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:10.391522 8711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:10.391916 8711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:10.395059 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "400f142c225a45c59a15414d6bcdb19e"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "78981ab83f75c032a33f414b723a3702"
server_key_iv: "23c375dfb54290028037be8faf123d05"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.395864 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "400f142c225a45c59a15414d6bcdb19e"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "78981ab83f75c032a33f414b723a3702"
server_key_iv: "23c375dfb54290028037be8faf123d05"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.401160 8711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:33:10.404660 8726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:10.406149 8711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:10.406324 8711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "400f142c225a45c59a15414d6bcdb19e"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "78981ab83f75c032a33f414b723a3702"
server_key_iv: "23c375dfb54290028037be8faf123d05"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.406540 8711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:10.464478 8711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:10.469441 8711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:10.469802 8711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:10.470865 8711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:10.472436 8711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:10.472553 8711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:10.472718 8711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:10.472796 8711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:10.508339 8711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40735
I20260927 18:33:10.508378 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40735 every 8 connection(s)
I20260927 18:33:10.509774 8711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:10.516860 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8711
I20260927 18:33:10.517087 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:33:10.517700 389 external_mini_cluster.cc:1428] Setting key 52b23092155fea1889156b6158101d28
Sep 27 18:33:10 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:10.529841 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:10.512574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54291 (local address 127.0.97.126:42537)
0927 18:33:10.513054 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:10.513061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:10.514189 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 18:33:10.516014 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:10.516019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:10.516081 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:10.516181 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:10.517811 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:10.518546 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:10.520359 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:10.520786 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:10.520934 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:10.524259 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:10.524264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:10.524267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:10.524295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:10.526861 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:10.527663 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:10.527671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:10.527674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:10.527790 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:10.528190 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:10.528197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:10.528202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:10.528437 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:10.528585 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:10.529260 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 18:33:10.529463 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260927 18:33:10.530763 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:10.513249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:54291)
0927 18:33:10.514030 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:10.514072 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:10.515735 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:10.516368 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:10.516375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:10.517013 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:10.517627 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:10.517643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:10.518711 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:10.518716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:10.520219 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:10.520234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:10.521032 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:10.521040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:10.521267 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:10.521983 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:10.522008 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:10.524054 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:10.527010 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:10.527018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:10.527034 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:10.527554 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:10.527896 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:10.527902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:10.527905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:10.528086 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:10.528564 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:10.528572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:10.528872 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 18:33:10.529131 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 18:33:10.529653 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":335,"threads_started":1}
I20260927 18:33:10.532212 8839 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42537
I20260927 18:33:10.532533 8839 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:10.533360 8839 heartbeater.cc:511] Master 127.0.97.126:42537 requested a full tablet report, sending...
I20260927 18:33:10.535948 8650 ts_manager.cc:194] Registered new tserver with Master: 400f142c225a45c59a15414d6bcdb19e (127.0.97.65:40735)
I20260927 18:33:10.538081 8650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:54291
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:33:10.567099 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:10.893206 8845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:10.893476 8845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:10.893627 8845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:10.900830 8845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:10.900925 8845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:10.901101 8845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:10.910892 8845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:10.912145 8845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:10.913476 8845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:10.926541 8851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:10.926621 8853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:10.926612 8850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:10.927476 8845 server_base.cc:1034] running on GCE node
I20260927 18:33:10.928079 8845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:10.929010 8845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:10.930245 8845 hybrid_clock.cc:648] HybridClock initialized: now 1790533990930197 us; error 61 us; skew 500 ppm
I20260927 18:33:10.930594 8845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:10 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:10.941710 8845 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:33:10.943200 8845 webserver.cc:460] Webserver started at http://127.0.97.66:34131/ using document root <none> and password file <none>
I20260927 18:33:10.943882 8845 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:10.944024 8845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:10.944339 8845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:10.947289 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "a6836f408480441097bb66a9ac55e024"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "8db7cb0e3c42aa7e44e51af97f52c81e"
server_key_iv: "c30a28aebd3b8de25ec46249a07b717a"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.948056 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "a6836f408480441097bb66a9ac55e024"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "8db7cb0e3c42aa7e44e51af97f52c81e"
server_key_iv: "c30a28aebd3b8de25ec46249a07b717a"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.953097 8845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:33:10.956617 8860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:10.957989 8845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:10.958145 8845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "a6836f408480441097bb66a9ac55e024"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "8db7cb0e3c42aa7e44e51af97f52c81e"
server_key_iv: "c30a28aebd3b8de25ec46249a07b717a"
server_key_version: "encryptionkey@0"
I20260927 18:33:10.958331 8845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:10.981055 8845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:10.985409 8845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:10.985699 8845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:10.986687 8845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:10.988054 8845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:10.988153 8845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:10.988302 8845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:10.988372 8845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:11.021864 8845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38819
I20260927 18:33:11.021917 8972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38819 every 8 connection(s)
I20260927 18:33:11.023166 8845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:11.027357 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8845
I20260927 18:33:11.027575 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:33:11.028097 389 external_mini_cluster.cc:1428] Setting key a79de124166880546ecf30d35578e234
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:11.044883 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.026060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44593 (local address 127.0.97.126:42537)
0927 18:33:11.026245 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:11.026249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:11.027795 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 18:33:11.030003 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:11.030008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:11.030088 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:11.030206 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:11.031701 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.032813 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.034381 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.035120 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.035250 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.039230 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:11.039235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:11.039238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:11.039268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:11.042027 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.042718 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.042729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.042732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.042846 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.043258 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.043263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.043266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.043530 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:11.043679 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:11.044337 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 18:33:11.044557 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:33:11.045989 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.026606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:44593)
0927 18:33:11.027608 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:11.027667 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:11.029698 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:11.030443 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:11.030452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:11.030930 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:11.031508 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:11.031535 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.032958 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.032962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:11.034241 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:11.034252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.035324 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.035329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:11.035554 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.036253 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:11.036274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:11.039013 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:11.042156 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:11.042165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:11.042181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:11.042595 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:11.042958 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:11.042963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:11.042966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:11.043141 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:11.043658 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:11.043665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:11.043987 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 18:33:11.044243 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 18:33:11.044734 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":385,"threads_started":1}
I20260927 18:33:11.047544 8973 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42537
I20260927 18:33:11.047835 8973 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:11.048542 8973 heartbeater.cc:511] Master 127.0.97.126:42537 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:33:11.050006 8650 ts_manager.cc:194] Registered new tserver with Master: a6836f408480441097bb66a9ac55e024 (127.0.97.66:38819)
I20260927 18:33:11.050786 8650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:44593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:33:11.077322 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:11.396687 8978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:11.396984 8978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:11.397146 8978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:11.404171 8978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:11.404311 8978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:11.404484 8978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:11.414037 8978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:11.415262 8978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:11.416618 8978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:11.429981 8983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:11.429945 8984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:11.429945 8986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:11.430846 8978 server_base.cc:1034] running on GCE node
I20260927 18:33:11.431396 8978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:11.432291 8978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:11.433573 8978 hybrid_clock.cc:648] HybridClock initialized: now 1790533991433536 us; error 27 us; skew 500 ppm
I20260927 18:33:11.433966 8978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:11.437127 8978 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:33:11.438602 8978 webserver.cc:460] Webserver started at http://127.0.97.67:45417/ using document root <none> and password file <none>
I20260927 18:33:11.439211 8978 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:11.439340 8978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:11.439661 8978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:11.442636 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "aa7c569bf3a74a3f9b2d5be4ab886184"
format_stamp: "Formatted at 2026-09-27 18:33:11 on dist-test-slave-n3p4"
server_key: "59eb49cea2d4411869bc514ac2b596f9"
server_key_iv: "ee2238c09c461598a8e4b88d0d4f4590"
server_key_version: "encryptionkey@0"
I20260927 18:33:11.443378 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "aa7c569bf3a74a3f9b2d5be4ab886184"
format_stamp: "Formatted at 2026-09-27 18:33:11 on dist-test-slave-n3p4"
server_key: "59eb49cea2d4411869bc514ac2b596f9"
server_key_iv: "ee2238c09c461598a8e4b88d0d4f4590"
server_key_version: "encryptionkey@0"
I20260927 18:33:11.448271 8978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:33:11.451761 8993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:11.453161 8978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:11.453315 8978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "aa7c569bf3a74a3f9b2d5be4ab886184"
format_stamp: "Formatted at 2026-09-27 18:33:11 on dist-test-slave-n3p4"
server_key: "59eb49cea2d4411869bc514ac2b596f9"
server_key_iv: "ee2238c09c461598a8e4b88d0d4f4590"
server_key_version: "encryptionkey@0"
I20260927 18:33:11.453503 8978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:11.473055 8978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:11.477346 8978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:11.477656 8978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:11.478670 8978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:11.479964 8978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:11.480036 8978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:11.480176 8978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:11.480224 8978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:11.514204 8978 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35121
I20260927 18:33:11.514226 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35121 every 8 connection(s)
I20260927 18:33:11.515729 8978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:11.516830 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 8978
I20260927 18:33:11.517050 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:33:11.517555 389 external_mini_cluster.cc:1428] Setting key 73c163e488fe6b3243967b60e89fbcd3
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:11.535252 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.518495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34341 (local address 127.0.97.126:42537)
0927 18:33:11.518860 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:11.518866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:11.520189 (+ 1323us) server_negotiation.cc:407] Connection header received
0927 18:33:11.522096 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:11.522100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:11.522184 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:11.522270 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:11.523632 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.524368 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.525822 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.526247 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.526343 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.529609 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:11.529614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:11.529617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:11.529649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:11.532374 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.533114 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.533121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.533124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.533244 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.533633 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.533640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.533643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.533948 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:11.534096 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:11.534710 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 18:33:11.534912 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260927 18:33:11.536120 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.519223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:34341)
0927 18:33:11.520040 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:11.520083 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:11.521863 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:11.522459 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:11.522467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:11.522969 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:11.523492 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:11.523508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.524490 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.524493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:11.525666 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:11.525676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.526441 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.526446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:11.526686 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.527305 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:11.527330 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:11.529438 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:11.532545 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:11.532554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:11.532570 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:11.532999 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:11.533350 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:11.533356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:11.533359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:11.533529 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:11.534070 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:11.534077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:11.534365 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 18:33:11.534620 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 18:33:11.535103 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":340,"threads_started":1}
I20260927 18:33:11.537302 9106 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42537
I20260927 18:33:11.537590 9106 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:11.538326 9106 heartbeater.cc:511] Master 127.0.97.126:42537 requested a full tablet report, sending...
I20260927 18:33:11.539652 8650 ts_manager.cc:194] Registered new tserver with Master: aa7c569bf3a74a3f9b2d5be4ab886184 (127.0.97.67:35121)
I20260927 18:33:11.540377 8650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:34341
I20260927 18:33:11.541190 8839 heartbeater.cc:503] Master 127.0.97.126:42537 was elected leader, sending a full tablet report...
I20260927 18:33:11.544775 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab.
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab.
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260927 18:33:11.674345 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8711
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:33:11.690533 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.675501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.97.65:40735)
0927 18:33:11.676265 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:11.676272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:11.676300 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:33:11.676360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:11.676366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:11.676591 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:11.676721 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:11.678068 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.679072 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.680241 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.680813 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.680906 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.684848 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:11.684858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:11.684873 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:11.684907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:11.687826 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.688490 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.688497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.688499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.688629 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.688963 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.688966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.688968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.689267 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:11.689484 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:11.689903 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 18:33:11.690124 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":447,"threads_started":1}
I20260927 18:33:11.691780 8834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:11.774863 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8711
I20260927 18:33:11.791152 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8845
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:33:11.806887 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.792048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.0.97.66:38819)
0927 18:33:11.792660 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:11.792667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:11.792699 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:33:11.792756 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:11.792762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:11.792975 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:11.793144 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:11.794456 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.795441 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.796744 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.797295 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.797437 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.801494 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:11.801505 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:11.801519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:11.801557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:11.804199 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.804837 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.804844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.804846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.804980 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.805338 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.805345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.805348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.805665 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:11.805998 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:11.806285 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:33:11.806538 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":336,"threads_started":1}
I20260927 18:33:11.808177 8968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:11.892902 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8845
I20260927 18:33:11.908595 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8978
Sep 27 18:33:11 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:33:11.924845 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:11.909483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.0.97.67:35121)
0927 18:33:11.910099 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:11.910106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:11.910134 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:33:11.910195 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:11.910202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:11.910403 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:11.910591 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:11.911868 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.912854 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.914256 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:11.914798 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:11.914946 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:11.919502 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:11.919512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:11.919526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:11.919557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:11.922471 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.923113 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.923119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.923122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.923230 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:11.923556 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:11.923559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:11.923561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:11.923859 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:11.924093 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:11.924367 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 18:33:11.924570 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":296,"threads_started":1}
I20260927 18:33:11.926254 9101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:12.007652 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8978
I20260927 18:33:12.023973 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 8618
I20260927 18:33:12.025254 8680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:12.126677 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 8618
I20260927 18:33:12.147100 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42537
--webserver_interface=127.0.97.126
--webserver_port=39021
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:12.460810 9123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:12.461139 9123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:12.461342 9123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:12.468346 9123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:12.468493 9123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:12.468611 9123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:12.468705 9123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:12.468798 9123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:12.478354 9123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=39021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:12.479615 9123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:12.480892 9123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:12.492436 9128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:12.492403 9131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:12.492352 9129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:12.493695 9123 server_base.cc:1034] running on GCE node
I20260927 18:33:12.494565 9123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:12.495970 9123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:12.497285 9123 hybrid_clock.cc:648] HybridClock initialized: now 1790533992497223 us; error 51 us; skew 500 ppm
I20260927 18:33:12.497632 9123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:12 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:12.500888 9123 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:33:12.502276 9123 webserver.cc:460] Webserver started at http://127.0.97.126:39021/ using document root <none> and password file <none>
I20260927 18:33:12.502900 9123 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:12.503041 9123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:12.508301 9123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:33:12.511171 9138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:12.512499 9123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:12.512692 9123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "d2aeaf1a1351439792a2d2dab191ba99"
format_stamp: "Formatted at 2026-09-27 18:33:09 on dist-test-slave-n3p4"
server_key: "2b29ce2a53c48c34ed69dddd7f7d9a15"
server_key_iv: "e92f9bd2cf7feeac238e0f3c48410b09"
server_key_version: "encryptionkey@0"
I20260927 18:33:12.513672 9123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:12.535936 9123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:12.540475 9123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:12.540752 9123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:12.559176 9123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42537
I20260927 18:33:12.559230 9189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42537 every 8 connection(s)
I20260927 18:33:12.560714 9123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:12.567611 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9123
I20260927 18:33:12.567813 9190 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:33:12.568421 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:40735
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:42537
--webserver_port=43769
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:33:12.571151 9190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Bootstrap starting.
I20260927 18:33:12.596604 9190 log.cc:824] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:12.607445 9190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:33:12.608021 9190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Bootstrap complete.
I20260927 18:33:12.618921 9190 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:12.619748 9190 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2aeaf1a1351439792a2d2dab191ba99, State: Initialized, Role: FOLLOWER
I20260927 18:33:12.620393 9190 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:12.620638 9190 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:12.620769 9190 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:12.620961 9190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:33:12.624636 9190 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:12.625113 9190 leader_election.cc:302] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2aeaf1a1351439792a2d2dab191ba99; no voters:
I20260927 18:33:12.625756 9190 leader_election.cc:288] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:33:12.625917 9194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:33:12.627187 9194 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [term 2 LEADER]: Becoming Leader. State: Replica: d2aeaf1a1351439792a2d2dab191ba99, State: Running, Role: LEADER
I20260927 18:33:12.627760 9194 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } }
I20260927 18:33:12.628125 9190 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:12.630273 9195 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2aeaf1a1351439792a2d2dab191ba99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } } }
I20260927 18:33:12.630703 9195 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:12.633038 9196 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2aeaf1a1351439792a2d2dab191ba99. Latest consensus state: current_term: 2 leader_uuid: "d2aeaf1a1351439792a2d2dab191ba99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeaf1a1351439792a2d2dab191ba99" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42537 } } }
I20260927 18:33:12.633476 9196 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:12.635231 9200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:12.641793 9200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:12.643518 9200 catalog_manager.cc:1259] Loaded cluster ID: 763366c6325a4619bd9cf73ed9613963
I20260927 18:33:12.643671 9200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:12.648908 9200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:12.650661 9200 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2aeaf1a1351439792a2d2dab191ba99: Loaded TSK: 0
I20260927 18:33:12.652709 9200 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:33:12.930429 9192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:12.930742 9192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:12.930950 9192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:12.938319 9192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:12.938473 9192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:12.938648 9192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:12.948490 9192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:40735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=43769
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:12.949918 9192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:12.951259 9192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:12.964819 9217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:12.965322 9216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:12.966780 9219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:12.967674 9192 server_base.cc:1034] running on GCE node
I20260927 18:33:12.968304 9192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:12.969244 9192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:12.970479 9192 hybrid_clock.cc:648] HybridClock initialized: now 1790533992970425 us; error 47 us; skew 500 ppm
I20260927 18:33:12.970865 9192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:12 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533992, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:12.974100 9192 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260927 18:33:12.975593 9192 webserver.cc:460] Webserver started at http://127.0.97.65:43769/ using document root <none> and password file <none>
I20260927 18:33:12.976289 9192 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:12.976470 9192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:12.981920 9192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:33:12.984663 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:12.986181 9192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:33:12.986352 9192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "400f142c225a45c59a15414d6bcdb19e"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "78981ab83f75c032a33f414b723a3702"
server_key_iv: "23c375dfb54290028037be8faf123d05"
server_key_version: "encryptionkey@0"
I20260927 18:33:12.987293 9192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:13.031776 9192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:13.036310 9192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:13.036636 9192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:13.037616 9192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:13.039003 9192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:13.039124 9192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.039254 9192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:13.039347 9192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.072815 9192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40735
I20260927 18:33:13.072912 9338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40735 every 8 connection(s)
I20260927 18:33:13.074415 9192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:13.084101 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9192
I20260927 18:33:13.084774 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:38819
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:42537
--webserver_port=34131
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:13.093647 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.077646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:44981)
0927 18:33:13.078533 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.078571 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.080266 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.081627 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.081635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.082210 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.082760 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.082777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.084971 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.084979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.086437 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.086450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.087996 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.088003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.088224 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.089021 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.089044 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.092765 (+ 3721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":419,"threads_started":1}
W20260927 18:33:13.094472 9339 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42537 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:42537: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
W20260927 18:33:13.094471 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.077594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44981 (local address 127.0.97.126:42537)
0927 18:33:13.078403 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.078419 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.078697 (+ 278us) server_negotiation.cc:407] Connection header received
0927 18:33:13.080781 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.080794 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.081130 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.081417 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.082961 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.084761 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.086587 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.087763 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.087966 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.093216 (+ 5250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:44981: BlockingRecv error: recv got EOF from 127.0.97.65:44981 (error 108)
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":335,"threads_started":1}
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:13.102841 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.095174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:44213)
0927 18:33:13.095355 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.095370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.095444 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.095824 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.095829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.096144 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.096269 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.096276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.097260 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.097265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.098198 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.098208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.098891 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.098897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.099061 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.099677 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.099692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.102556 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:33:13.102959 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.095418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44213 (local address 127.0.97.126:42537)
0927 18:33:13.095604 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.095608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.095621 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:33:13.095656 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.095659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.095707 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.095805 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.096397 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.097125 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.098372 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.098756 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.098881 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.102726 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:44213: BlockingRecv error: recv got EOF from 127.0.97.65:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:13.111181 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.103440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:33325)
0927 18:33:13.103588 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.103604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.103697 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.104181 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.104186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.104629 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.104802 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.104814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.105742 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.105747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.106727 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.106738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.107412 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.107418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.107633 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.108220 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.108235 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.110888 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:33:13.111428 9339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:33:13.111577 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.103779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33325 (local address 127.0.97.126:42537)
0927 18:33:13.103944 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.103947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.103960 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:33:13.103994 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.103997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.104066 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.104162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.104947 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.105602 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.106863 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.107273 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.107404 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.111082 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:33325: BlockingRecv error: recv got EOF from 127.0.97.65:33325 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:33:13.412382 9343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:13.412705 9343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:13.412873 9343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:13.420262 9343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:13.420418 9343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:13.420588 9343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:13.430238 9343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:38819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=34131
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:13.431588 9343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:13.432843 9343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:13.445977 9351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:13.446046 9349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:13.446046 9348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:13.446909 9343 server_base.cc:1034] running on GCE node
I20260927 18:33:13.447531 9343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:13.448457 9343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:13.449689 9343 hybrid_clock.cc:648] HybridClock initialized: now 1790533993449640 us; error 42 us; skew 500 ppm
I20260927 18:33:13.450062 9343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533993, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:13.453258 9343 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260927 18:33:13.454778 9343 webserver.cc:460] Webserver started at http://127.0.97.66:34131/ using document root <none> and password file <none>
I20260927 18:33:13.455437 9343 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:13.455559 9343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:13.460907 9343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:33:13.463860 9358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.465363 9343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:13.465592 9343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "a6836f408480441097bb66a9ac55e024"
format_stamp: "Formatted at 2026-09-27 18:33:10 on dist-test-slave-n3p4"
server_key: "8db7cb0e3c42aa7e44e51af97f52c81e"
server_key_iv: "c30a28aebd3b8de25ec46249a07b717a"
server_key_version: "encryptionkey@0"
I20260927 18:33:13.466683 9343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:13.495077 9343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:13.499778 9343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:13.500082 9343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:13.501106 9343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:13.502562 9343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:13.502661 9343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.502820 9343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:13.502907 9343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.538739 9343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38819
I20260927 18:33:13.538842 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38819 every 8 connection(s)
I20260927 18:33:13.540277 9343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:13.547181 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9343
I20260927 18:33:13.547791 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:35121
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:42537
--webserver_port=45417
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:33:13.556437 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.542977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34551 (local address 127.0.97.126:42537)
0927 18:33:13.543144 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.543148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.544487 (+ 1339us) server_negotiation.cc:407] Connection header received
0927 18:33:13.546562 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.546568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.546655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.546763 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.548361 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.549053 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.550655 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.551068 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.551213 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.556197 (+ 4984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:34551: BlockingRecv error: recv got EOF from 127.0.97.66:34551 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:33:13.557154 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.543420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:34551)
0927 18:33:13.544313 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.544359 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.546224 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.547011 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.547019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.547572 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.548192 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.548211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.549205 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.549217 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.550497 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.550517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.551293 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.551300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.551537 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.552245 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.552266 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.556030 (+ 3764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
W20260927 18:33:13.557829 9471 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42537 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:42537: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:13.566493 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.558414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:52067)
0927 18:33:13.558656 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.558675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.558782 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.559107 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.559111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.559434 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.559573 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.559580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.560620 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.560627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.561584 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.561595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.562558 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.562563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.562704 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.563280 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.563297 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.566213 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 18:33:13.566668 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.558464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52067 (local address 127.0.97.126:42537)
0927 18:33:13.558614 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.558620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.558728 (+ 108us) server_negotiation.cc:407] Connection header received
0927 18:33:13.558920 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.558924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.558979 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.559085 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.559705 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.560493 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.561712 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.562216 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.562369 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.566403 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:52067: BlockingRecv error: recv got EOF from 127.0.97.66:52067 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:13.575323 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:13.567386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:57273)
0927 18:33:13.567614 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:13.567632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:13.567731 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:13.568060 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:13.568064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:13.568355 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:13.568485 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.568492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.569507 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.569512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.570680 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:13.570691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.571469 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.571475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:13.571672 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.572317 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:13.572334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:13.575032 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260927 18:33:13.575450 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:13.567443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57273 (local address 127.0.97.126:42537)
0927 18:33:13.567611 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:13.567617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:13.567679 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:33:13.567865 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:13.567870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:13.567921 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:13.568041 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:13.568624 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.569369 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.570805 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:13.571336 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:13.571469 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:13.575229 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:57273: BlockingRecv error: recv got EOF from 127.0.97.66:57273 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:33:13.575824 9471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:33:13.876971 9474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:13.877247 9474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:13.877404 9474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:13.886152 9474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:13.886261 9474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:13.886459 9474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:13.896528 9474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:35121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=45417
--tserver_master_addrs=127.0.97.126:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:13.897951 9474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:13.899225 9474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:13.912670 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:13.912698 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:13.912650 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:13.913895 9474 server_base.cc:1034] running on GCE node
I20260927 18:33:13.914577 9474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:13.915494 9474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:13.916778 9474 hybrid_clock.cc:648] HybridClock initialized: now 1790533993916718 us; error 46 us; skew 500 ppm
I20260927 18:33:13.917104 9474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:13 dist-test-slave-n3p4 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790533993, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:13.920203 9474 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260927 18:33:13.921685 9474 webserver.cc:460] Webserver started at http://127.0.97.67:45417/ using document root <none> and password file <none>
I20260927 18:33:13.922392 9474 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:13.922537 9474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:13.927678 9474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:33:13.930397 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.931687 9474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:13.931813 9474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "aa7c569bf3a74a3f9b2d5be4ab886184"
format_stamp: "Formatted at 2026-09-27 18:33:11 on dist-test-slave-n3p4"
server_key: "59eb49cea2d4411869bc514ac2b596f9"
server_key_iv: "ee2238c09c461598a8e4b88d0d4f4590"
server_key_version: "encryptionkey@0"
I20260927 18:33:13.932663 9474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:13.962771 9474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:13.967042 9474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:13.967263 9474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:13.968134 9474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:13.969340 9474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:13.969422 9474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:13.969488 9474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:13.969540 9474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:14.002686 9474 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35121
I20260927 18:33:14.002774 9601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35121 every 8 connection(s)
I20260927 18:33:14.004065 9474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:14.011492 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9474
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:33:14.020947 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:14.006500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47757 (local address 127.0.97.126:42537)
0927 18:33:14.006678 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.006682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.007713 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 18:33:14.009521 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.009527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.009617 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.009728 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.011278 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.012109 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.014477 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.014875 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.014978 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.020736 (+ 5758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:47757: BlockingRecv error: recv got EOF from 127.0.97.67:47757 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:33:14.021741 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.006882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:47757)
0927 18:33:14.007556 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:14.007592 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:14.009245 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:14.010001 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:14.010008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:14.010518 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:14.011089 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.011114 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.012275 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.012282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.014332 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.014349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.015103 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.015114 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.015305 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.015993 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:14.016025 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:14.020598 (+ 4573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
W20260927 18:33:14.022516 9602 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42537 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:42537: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
I20260927 18:33:14.028258 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.012535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.97.126:42537)
0927 18:33:14.013228 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.013232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.013245 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:33:14.013281 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.013284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.013331 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.013411 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.015128 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.015833 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.017593 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.018212 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.018352 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.021326 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:14.021333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:14.021348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:14.021383 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:14.025003 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:14.025670 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:14.025676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:14.025682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:14.025922 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:14.026299 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:14.026306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:14.026309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:14.026740 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:14.027032 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:14.027488 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 18:33:14.027838 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":487,"threads_started":1}
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:14.030941 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.023035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:55053)
0927 18:33:14.023225 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:14.023244 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:14.023345 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:14.023641 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:14.023646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:14.024032 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:14.024191 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.024201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.025160 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.025166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.026132 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.026144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.026838 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.026845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.027004 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.027555 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:14.027574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:14.030696 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:33:14.031502 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:14.023160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55053 (local address 127.0.97.126:42537)
0927 18:33:14.023280 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.023284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.023298 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:33:14.023445 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.023448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.023499 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.023572 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.024312 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.025019 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.026271 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.026697 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.026824 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.031262 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:55053: BlockingRecv error: recv got EOF from 127.0.97.67:55053 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:14.039182 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.031597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:50963)
0927 18:33:14.032118 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:14.032132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:14.032251 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:14.032542 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:14.032547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:14.032833 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:14.032971 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.032978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.033978 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.033984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.034884 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.034894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.035654 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.035660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.035893 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.036397 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:14.036412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:14.038994 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439}
W20260927 18:33:14.039345 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:14.031794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50963 (local address 127.0.97.126:42537)
0927 18:33:14.031963 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.031966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.032200 (+ 234us) server_negotiation.cc:407] Connection header received
0927 18:33:14.032350 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.032355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.032412 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.032523 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.033098 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.033834 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.035061 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.035521 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.035645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.039154 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:50963: BlockingRecv error: recv got EOF from 127.0.97.67:50963 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 18:33:14.039530 9602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:14.120602 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.111985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:34331)
0927 18:33:14.112464 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:14.112480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:14.112596 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:14.112882 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:14.112885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:14.113379 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:14.113576 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.113606 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.114874 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.114886 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.115924 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.115943 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.116745 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.116751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.116920 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.117512 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:14.117529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:14.120383 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
W20260927 18:33:14.120807 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:14.112099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34331 (local address 127.0.97.126:42537)
0927 18:33:14.112339 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.112346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.112564 (+ 218us) server_negotiation.cc:407] Connection header received
0927 18:33:14.112696 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.112700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.112759 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.112840 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.113751 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.114734 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.116072 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.116571 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.116709 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.120580 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:34331: BlockingRecv error: recv got EOF from 127.0.97.65:34331 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:14 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:14.585175 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:14.576389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:52855)
0927 18:33:14.576904 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:14.576921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:14.577033 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:14.577345 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:14.577348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:14.577912 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:14.578099 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.578107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.579146 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.579151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.580137 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:14.580149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.581117 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.581123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:14.581269 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.582095 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:14.582120 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:14.584860 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
W20260927 18:33:14.585309 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:14.576552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52855 (local address 127.0.97.126:42537)
0927 18:33:14.576764 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:14.576771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:14.577002 (+ 231us) server_negotiation.cc:407] Connection header received
0927 18:33:14.577135 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:14.577139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:14.577201 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:14.577297 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:14.578251 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.578995 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.580303 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:14.580915 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:14.581055 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:14.585051 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:52855: BlockingRecv error: recv got EOF from 127.0.97.66:52855 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:15.048295 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:15.040112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:42053)
0927 18:33:15.040579 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:15.040604 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:15.040710 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:15.040983 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:15.040986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:15.041417 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:15.041635 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.041643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.042859 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.042864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.043868 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.043880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.044562 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.044566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.044681 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.045248 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:15.045265 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:15.048070 (+ 2805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
W20260927 18:33:15.048485 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:15.040154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42053 (local address 127.0.97.126:42537)
0927 18:33:15.040360 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:15.040369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:15.040704 (+ 335us) server_negotiation.cc:407] Connection header received
0927 18:33:15.040793 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:15.040797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:15.040857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:15.040940 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:15.041804 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.042705 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.044003 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.044401 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.044493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.048253 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:42053: BlockingRecv error: recv got EOF from 127.0.97.67:42053 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:15.130653 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:15.121557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:60919)
0927 18:33:15.122121 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:15.122138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:15.122249 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:15.122661 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:15.122666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:15.123161 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:15.123309 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.123319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.124408 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.124417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.125435 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.125449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.126270 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.126277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.126466 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.127196 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:15.127214 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:15.130304 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":334,"threads_started":1}
W20260927 18:33:15.130743 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:15.121587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60919 (local address 127.0.97.126:42537)
0927 18:33:15.121816 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:15.121823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:15.122232 (+ 409us) server_negotiation.cc:407] Connection header received
0927 18:33:15.122429 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:15.122434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:15.122497 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:15.122618 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:15.123471 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.124271 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.125585 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.126096 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.126252 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.130494 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:60919: BlockingRecv error: recv got EOF from 127.0.97.65:60919 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:15 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:15.594214 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:15.586255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:56183)
0927 18:33:15.586761 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:15.586776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:15.586881 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:15.587176 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:15.587179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:15.587555 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:15.587688 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.587695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.588719 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.588722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.589608 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:15.589618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.590382 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.590386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:15.590505 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.591166 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:15.591183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:15.593979 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":300,"threads_started":1}
W20260927 18:33:15.594522 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:15.586245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56183 (local address 127.0.97.126:42537)
0927 18:33:15.586463 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:15.586471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:15.586867 (+ 396us) server_negotiation.cc:407] Connection header received
0927 18:33:15.586980 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:15.586984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:15.587046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:15.587130 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:15.587837 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.588594 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.589740 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:15.590245 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:15.590358 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:15.594208 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:56183: BlockingRecv error: recv got EOF from 127.0.97.66:56183 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:16.061123 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:16.049392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:55623)
0927 18:33:16.049975 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:16.049990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:16.050146 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:16.050537 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:16.050542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:16.050940 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:16.051078 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.051086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.052179 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.052186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.053090 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.053101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.053835 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.053842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.054009 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.058007 (+ 3998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:16.058025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:16.060782 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
W20260927 18:33:16.061244 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:16.049645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55623 (local address 127.0.97.126:42537)
0927 18:33:16.049869 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:16.049876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:16.050074 (+ 198us) server_negotiation.cc:407] Connection header received
0927 18:33:16.050269 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:16.050274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:16.050343 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:16.050447 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:16.051231 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.052042 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.053227 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.053653 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.053779 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.060985 (+ 7206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:55623: BlockingRecv error: recv got EOF from 127.0.97.67:55623 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:16.140226 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:16.131707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:60195)
0927 18:33:16.132199 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:16.132214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:16.132365 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:16.132716 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:16.132721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:16.133089 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:16.133243 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.133252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.134399 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.134407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.135658 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.135670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.136340 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.136346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.136513 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.137176 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:16.137193 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:16.139858 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":253,"threads_started":1}
W20260927 18:33:16.140280 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:16.131759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60195 (local address 127.0.97.126:42537)
0927 18:33:16.132000 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:16.132007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:16.132299 (+ 292us) server_negotiation.cc:407] Connection header received
0927 18:33:16.132491 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:16.132495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:16.132567 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:16.132670 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:16.133389 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.134263 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.135811 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.136202 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.136331 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.140039 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:60195: BlockingRecv error: recv got EOF from 127.0.97.65:60195 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:16 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:16.604034 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:16.595261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:32887)
0927 18:33:16.595853 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:16.595870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:16.595988 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:16.596337 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:16.596340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:16.596859 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:16.597034 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.597043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.598180 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.598183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.599108 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:16.599119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.599761 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.599765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:16.599883 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.600611 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:16.600628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:16.603719 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":345,"threads_started":1}
W20260927 18:33:16.604205 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:16.595317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:32887 (local address 127.0.97.126:42537)
0927 18:33:16.595508 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:16.595512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:16.595964 (+ 452us) server_negotiation.cc:407] Connection header received
0927 18:33:16.596132 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:16.596136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:16.596200 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:16.596291 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:16.597192 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.598041 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.599240 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:16.599637 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:16.599739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:16.603946 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:32887: BlockingRecv error: recv got EOF from 127.0.97.66:32887 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:17.070742 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:17.062258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:35309)
0927 18:33:17.062757 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:17.062772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:17.062929 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:17.063291 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:17.063296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:17.063660 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:17.063811 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.063820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.064819 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.064826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.065739 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.065750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.066469 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.066475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.066644 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.067266 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:17.067283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:17.070392 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":286,"threads_started":1}
W20260927 18:33:17.070788 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:17.062343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35309 (local address 127.0.97.126:42537)
0927 18:33:17.062565 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:17.062573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:17.062859 (+ 286us) server_negotiation.cc:407] Connection header received
0927 18:33:17.063058 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:17.063063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:17.063156 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:17.063259 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:17.063967 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.064682 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.065898 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.066317 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.066449 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.070551 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:35309: BlockingRecv error: recv got EOF from 127.0.97.67:35309 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:17.150542 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:17.141528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:53877)
0927 18:33:17.142188 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:17.142205 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:17.142324 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:17.142678 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:17.142685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:17.143185 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:17.143403 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.143415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.144874 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.144879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.145919 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.145930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.146655 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.146660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.146784 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.147384 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:17.147400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:17.150293 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":407,"threads_started":1}
W20260927 18:33:17.150691 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:17.141565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53877 (local address 127.0.97.126:42537)
0927 18:33:17.141887 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:17.141910 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:17.142295 (+ 385us) server_negotiation.cc:407] Connection header received
0927 18:33:17.142438 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:17.142442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:17.142506 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:17.142592 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:17.143575 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.144744 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.146066 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.146524 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.146627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.150480 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:53877: BlockingRecv error: recv got EOF from 127.0.97.65:53877 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:17 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:17.613310 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:17.604941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:51495)
0927 18:33:17.605512 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:17.605528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:17.605646 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:17.605957 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:17.605959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:17.606313 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:17.606487 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.606496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.607464 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.607467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.608570 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:17.608585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.609532 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.609540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:17.609738 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.610385 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:17.610434 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:17.613080 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":335,"threads_started":1}
W20260927 18:33:17.613587 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:17.605002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51495 (local address 127.0.97.126:42537)
0927 18:33:17.605166 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:17.605171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:17.605629 (+ 458us) server_negotiation.cc:407] Connection header received
0927 18:33:17.605751 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:17.605755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:17.605834 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:17.605924 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:17.606628 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.607340 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.608710 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:17.609354 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:17.609485 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:17.613304 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:51495: BlockingRecv error: recv got EOF from 127.0.97.66:51495 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:18.080288 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:18.071841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:58569)
0927 18:33:18.072326 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:18.072342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:18.072539 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:18.072916 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:18.072921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:18.073317 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:18.073475 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.073483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.074664 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.074672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.075633 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.075643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.076368 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.076374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.076559 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.077197 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:18.077215 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:18.080066 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
W20260927 18:33:18.080495 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:18.071870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58569 (local address 127.0.97.126:42537)
0927 18:33:18.072092 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:18.072100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:18.072432 (+ 332us) server_negotiation.cc:407] Connection header received
0927 18:33:18.072666 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:18.072672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:18.072762 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:18.072882 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:18.073622 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.074532 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.075773 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.076221 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.076357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.080251 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:58569: BlockingRecv error: recv got EOF from 127.0.97.67:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:18.159883 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:18.151542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:40689)
0927 18:33:18.152054 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:18.152072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:18.152189 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:18.152517 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:18.152520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:18.152921 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:18.153072 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.153080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.154226 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.154230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.155197 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.155208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.155863 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.155868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.155991 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.156630 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:18.156655 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:18.159535 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
W20260927 18:33:18.159977 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:18.151625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40689 (local address 127.0.97.126:42537)
0927 18:33:18.151869 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:18.151876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:18.152151 (+ 275us) server_negotiation.cc:407] Connection header received
0927 18:33:18.152287 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:18.152290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:18.152347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:18.152433 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:18.153231 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.154061 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.155327 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.155734 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.155827 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.159720 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:40689: BlockingRecv error: recv got EOF from 127.0.97.65:40689 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:18 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:18.622830 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:18.614350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:43199)
0927 18:33:18.614892 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:18.614908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:18.615054 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:18.615460 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:18.615466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:18.615906 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:18.616085 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.616093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.617153 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.617164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.618338 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:18.618349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.619063 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.619067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:18.619189 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.619807 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:18.619823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:18.622581 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":274,"threads_started":1}
W20260927 18:33:18.623063 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:18.614486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43199 (local address 127.0.97.126:42537)
0927 18:33:18.614658 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:18.614664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:18.614996 (+ 332us) server_negotiation.cc:407] Connection header received
0927 18:33:18.615203 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:18.615208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:18.615291 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:18.615390 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:18.616214 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.616963 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.618480 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:18.618929 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:18.619028 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:18.622780 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:43199: BlockingRecv error: recv got EOF from 127.0.97.66:43199 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:19.089630 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:19.081304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:58791)
0927 18:33:19.081862 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:19.081888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:19.082037 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:19.082390 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:19.082436 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:19.082857 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:19.083013 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.083021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.083998 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.084006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.084926 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.084937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.085634 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.085640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.085869 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.086500 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:19.086516 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:19.089361 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":262,"threads_started":1}
W20260927 18:33:19.089709 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:19.081530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58791 (local address 127.0.97.126:42537)
0927 18:33:19.081707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:19.081715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:19.081977 (+ 262us) server_negotiation.cc:407] Connection header received
0927 18:33:19.082188 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:19.082193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:19.082259 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:19.082359 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:19.083153 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.083856 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.085062 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.085484 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.085614 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.089536 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:58791: BlockingRecv error: recv got EOF from 127.0.97.67:58791 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:19.169210 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:19.160806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:41873)
0927 18:33:19.161454 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:19.161472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:19.161606 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:19.161908 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:19.161912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:19.162414 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:19.162594 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.162604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.163735 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.163742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.164672 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.164684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.165427 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.165434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.165626 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.166301 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:19.166319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:19.168980 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":390,"threads_started":1}
W20260927 18:33:19.169468 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:19.160897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41873 (local address 127.0.97.126:42537)
0927 18:33:19.161262 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:19.161268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:19.161564 (+ 296us) server_negotiation.cc:407] Connection header received
0927 18:33:19.161706 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:19.161709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:19.161765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:19.161856 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:19.162752 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.163582 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.164831 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.165261 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.165411 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.169190 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:41873: BlockingRecv error: recv got EOF from 127.0.97.65:41873 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:19 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:19.632220 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:19.623713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:39675)
0927 18:33:19.624232 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:19.624247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:19.624394 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:19.624722 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:19.624726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:19.625125 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:19.625271 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.625279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.626316 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.626322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.627214 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:19.627224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.627961 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.627967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:19.628165 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.628869 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:19.628888 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:19.631875 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":298,"threads_started":1}
W20260927 18:33:19.632444 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:19.623800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39675 (local address 127.0.97.126:42537)
0927 18:33:19.624044 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:19.624053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:19.624331 (+ 278us) server_negotiation.cc:407] Connection header received
0927 18:33:19.624526 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:19.624531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:19.624596 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:19.624693 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:19.625435 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.626192 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.627363 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:19.627777 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:19.627918 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:19.632138 (+ 4220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:39675: BlockingRecv error: recv got EOF from 127.0.97.66:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:20.098980 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:20.090666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:39855)
0927 18:33:20.091227 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:20.091247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:20.091395 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:20.091715 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:20.091718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:20.092164 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:20.092315 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.092323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.093335 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.093338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.094260 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.094272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.094987 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.094991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.095105 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.095690 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:20.095707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:20.098608 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":332,"threads_started":1}
W20260927 18:33:20.099073 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:20.090827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:39855 (local address 127.0.97.126:42537)
0927 18:33:20.090980 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:20.090985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:20.091324 (+ 339us) server_negotiation.cc:407] Connection header received
0927 18:33:20.091528 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:20.091532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:20.091594 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:20.091677 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:20.092471 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.093218 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.094401 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.094829 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.094943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.098810 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:39855: BlockingRecv error: recv got EOF from 127.0.97.67:39855 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:20.178857 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:20.170107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:34767)
0927 18:33:20.170849 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:20.170872 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:20.171017 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:20.171372 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:20.171376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:20.171929 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:20.172099 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.172108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.173117 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.173124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.174094 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.174106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.174800 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.174806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.174975 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.175684 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:20.175702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:20.178628 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":294,"threads_started":1}
W20260927 18:33:20.179169 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:20.170195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34767 (local address 127.0.97.126:42537)
0927 18:33:20.170433 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:20.170443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:20.170960 (+ 517us) server_negotiation.cc:407] Connection header received
0927 18:33:20.171164 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:20.171169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:20.171242 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:20.171341 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:20.172248 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.172978 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.174233 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.174649 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.174780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.178844 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:34767: BlockingRecv error: recv got EOF from 127.0.97.65:34767 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:20 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:20.641701 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:20.633193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:50881)
0927 18:33:20.633705 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:20.633720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:20.633943 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:20.634317 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:20.634324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:20.634727 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:20.634886 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.634894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.636096 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.636101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.637085 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:20.637097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.637813 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.637820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:20.637989 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.638622 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:20.638640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:20.641343 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":223,"threads_started":1}
W20260927 18:33:20.641798 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:20.633230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50881 (local address 127.0.97.126:42537)
0927 18:33:20.633483 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:20.633495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:20.633805 (+ 310us) server_negotiation.cc:407] Connection header received
0927 18:33:20.634067 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:20.634073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:20.634168 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:20.634271 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:20.635086 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.635945 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.637241 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:20.637634 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:20.637766 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:20.641526 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:50881: BlockingRecv error: recv got EOF from 127.0.97.66:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:21.112748 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:21.099980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:43103)
0927 18:33:21.100571 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:21.100588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:21.100752 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:21.101181 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:21.101188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:21.101771 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:21.102026 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.102040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.103128 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.103133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.104118 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.104130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.104826 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.104831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.104953 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.109286 (+ 4333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:21.109303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:21.112420 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":271,"threads_started":1}
W20260927 18:33:21.112882 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:21.100073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43103 (local address 127.0.97.126:42537)
0927 18:33:21.100301 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:21.100310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:21.100700 (+ 390us) server_negotiation.cc:407] Connection header received
0927 18:33:21.100884 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:21.100890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:21.100968 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:21.101085 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:21.102192 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.102976 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.104248 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.104694 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.104793 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.112691 (+ 7898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:43103: BlockingRecv error: recv got EOF from 127.0.97.67:43103 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:21.188115 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:21.179857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:39713)
0927 18:33:21.180414 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:21.180436 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:21.180607 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:21.180944 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:21.180951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:21.181354 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:21.181518 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.181527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.182640 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.182646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.183529 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.183539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.184282 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.184288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.184464 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.185114 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:21.185131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:21.187765 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":279,"threads_started":1}
W20260927 18:33:21.188189 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:21.180148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39713 (local address 127.0.97.126:42537)
0927 18:33:21.180368 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:21.180376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:21.180533 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:33:21.180729 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:21.180734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:21.180798 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:21.180903 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:21.181658 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.182508 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.183671 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.184104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.184229 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.187952 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:39713: BlockingRecv error: recv got EOF from 127.0.97.65:39713 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:21 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:21.651226 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:21.642781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:56437)
0927 18:33:21.643343 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:21.643360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:21.643556 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:21.643904 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:21.643909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:21.644322 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:21.644478 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.644487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.645630 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.645636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.646610 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:21.646621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.647355 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.647361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:21.647536 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.648112 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:21.648129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:21.650903 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":316,"threads_started":1}
W20260927 18:33:21.651335 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:21.642894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56437 (local address 127.0.97.126:42537)
0927 18:33:21.643095 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:21.643103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:21.643479 (+ 376us) server_negotiation.cc:407] Connection header received
0927 18:33:21.643687 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:21.643692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:21.643767 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:21.643868 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:21.644622 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.645487 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.646750 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:21.647171 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:21.647298 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:21.651086 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:56437: BlockingRecv error: recv got EOF from 127.0.97.66:56437 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:22.122836 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:22.113933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:42015)
0927 18:33:22.114518 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:22.114536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:22.114652 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:22.114941 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:22.114945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:22.115341 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:22.115532 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.115540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.116515 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.116519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.117737 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.117754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.118675 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.118683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.118888 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.119596 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:22.119615 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:22.122481 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260927 18:33:22.122978 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:22.113965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42015 (local address 127.0.97.126:42537)
0927 18:33:22.114156 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:22.114160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:22.114608 (+ 448us) server_negotiation.cc:407] Connection header received
0927 18:33:22.114746 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:22.114750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:22.114808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:22.114887 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:22.115660 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.116373 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.117958 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.118476 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.118641 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.122673 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:42015: BlockingRecv error: recv got EOF from 127.0.97.67:42015 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:22.197206 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:22.189113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:42433)
0927 18:33:22.189739 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:22.189761 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:22.189918 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:22.190229 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:22.190232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:22.190630 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:22.190855 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.190867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.192003 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.192007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.192979 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.192990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.193622 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.193627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.193751 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.194311 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:22.194326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:22.196969 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":317,"threads_started":1}
W20260927 18:33:22.197414 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:22.189178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42433 (local address 127.0.97.126:42537)
0927 18:33:22.189394 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:22.189404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:22.189859 (+ 455us) server_negotiation.cc:407] Connection header received
0927 18:33:22.190042 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:22.190046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:22.190114 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:22.190191 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:22.190983 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.191884 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.193097 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.193488 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.193610 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.197175 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:42433: BlockingRecv error: recv got EOF from 127.0.97.65:42433 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:22 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:22.660961 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:22.652311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:36987)
0927 18:33:22.652969 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:22.652990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:22.653184 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:22.653588 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:22.653593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:22.654028 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:22.654187 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.654194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.655252 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.655256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.656253 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:22.656263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.656973 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.656977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:22.657102 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.657826 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:22.657873 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:22.660715 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":337,"threads_started":1}
W20260927 18:33:22.661202 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:22.652385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36987 (local address 127.0.97.126:42537)
0927 18:33:22.652624 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:22.652633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:22.653103 (+ 470us) server_negotiation.cc:407] Connection header received
0927 18:33:22.653314 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:22.653319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:22.653416 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:22.653521 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:22.654320 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.655140 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.656398 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:22.656822 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:22.656944 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:22.660967 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:36987: BlockingRecv error: recv got EOF from 127.0.97.66:36987 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:23.133246 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:23.123835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:43639)
0927 18:33:23.124334 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:23.124353 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:23.124550 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:23.124924 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:23.124932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:23.125363 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:23.125549 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.125558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.126713 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.126723 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.127739 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.127751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.128819 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.128829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.129090 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.129854 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:23.129872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:23.132979 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":262,"threads_started":1}
W20260927 18:33:23.133329 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:23.123879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43639 (local address 127.0.97.126:42537)
0927 18:33:23.124102 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:23.124109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:23.124489 (+ 380us) server_negotiation.cc:407] Connection header received
0927 18:33:23.124687 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:23.124694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:23.124765 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:23.124872 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:23.125698 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.126575 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.127936 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.128595 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.128732 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.133153 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:43639: BlockingRecv error: recv got EOF from 127.0.97.67:43639 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:23.207118 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:23.198200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:57361)
0927 18:33:23.198737 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:23.198766 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:23.198876 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:23.199204 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:23.199207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:23.199550 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:23.199681 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.199687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.200904 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.200909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.202289 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.202311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.203164 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.203170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.203344 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.203996 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:23.204013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:23.206771 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":325,"threads_started":1}
W20260927 18:33:23.207211 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:23.198258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57361 (local address 127.0.97.126:42537)
0927 18:33:23.198485 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:23.198489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:23.198842 (+ 353us) server_negotiation.cc:407] Connection header received
0927 18:33:23.198991 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:23.198995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:23.199079 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:23.199169 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:23.199802 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.200629 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.202475 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.203011 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.203146 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.206955 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:57361: BlockingRecv error: recv got EOF from 127.0.97.65:57361 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:23 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:23.669962 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:23.661949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:60515)
0927 18:33:23.662475 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:23.662495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:23.662627 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:23.662950 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:23.662954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:23.663309 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:23.663451 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.663458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.664454 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.664457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.665393 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:23.665403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.666132 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.666136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:23.666253 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.666838 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:23.666853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:23.669602 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
W20260927 18:33:23.670053 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:23.661986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60515 (local address 127.0.97.126:42537)
0927 18:33:23.662159 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:23.662167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:23.662601 (+ 434us) server_negotiation.cc:407] Connection header received
0927 18:33:23.662754 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:23.662759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:23.662824 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:23.662898 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:23.663580 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.664300 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.665516 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:23.665997 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:23.666085 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:23.669806 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:60515: BlockingRecv error: recv got EOF from 127.0.97.66:60515 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:24.142835 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:24.134255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:60263)
0927 18:33:24.134728 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:24.134745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:24.134853 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:24.135127 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:24.135130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:24.135585 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:24.135788 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.135797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.136768 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.136772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.137723 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.137734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.138436 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.138440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.138570 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.139131 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:24.139150 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:24.142506 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":284,"threads_started":1}
W20260927 18:33:24.142966 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:24.134427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60263 (local address 127.0.97.126:42537)
0927 18:33:24.134643 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:24.134650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:24.134819 (+ 169us) server_negotiation.cc:407] Connection header received
0927 18:33:24.134946 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:24.134949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:24.135005 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:24.135080 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:24.135924 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.136638 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.137900 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.138300 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.138394 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.142731 (+ 4337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:60263: BlockingRecv error: recv got EOF from 127.0.97.67:60263 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:24.216568 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:24.208143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:56881)
0927 18:33:24.208719 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:24.208744 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:24.208854 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:24.209262 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:24.209266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:24.209647 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:24.209847 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.209856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.210807 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.210811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.211720 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.211731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.212410 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.212414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.212537 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.213156 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:24.213172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:24.216331 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
W20260927 18:33:24.216759 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:24.208220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56881 (local address 127.0.97.126:42537)
0927 18:33:24.208400 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:24.208405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:24.208836 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:33:24.208996 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:24.209000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:24.209079 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:24.209168 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:24.209974 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.210693 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.211840 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.212282 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.212373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.216524 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56881: BlockingRecv error: recv got EOF from 127.0.97.65:56881 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:24 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:24.679898 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:24.670957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:55151)
0927 18:33:24.671589 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:24.671611 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:24.672010 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:24.672421 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:24.672425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:24.672832 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:24.673023 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.673032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.674231 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.674237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.675195 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:24.675206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.675923 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.675927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:24.676040 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.676654 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:24.676670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:24.679653 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":280,"threads_started":1}
W20260927 18:33:24.680128 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:24.671089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55151 (local address 127.0.97.126:42537)
0927 18:33:24.671326 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:24.671332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:24.671772 (+ 440us) server_negotiation.cc:407] Connection header received
0927 18:33:24.672182 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:24.672188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:24.672272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:24.672363 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:24.673151 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.674049 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.675319 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:24.675783 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:24.675879 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:24.679868 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:55151: BlockingRecv error: recv got EOF from 127.0.97.66:55151 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:25.152128 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:25.143733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:33873)
0927 18:33:25.144229 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:25.144245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:25.144357 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:25.144664 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:25.144667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:25.145025 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:25.145192 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.145200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.146303 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.146307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.147249 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.147261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.148041 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.148046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.148170 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.148854 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:25.148879 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:25.151820 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":274,"threads_started":1}
W20260927 18:33:25.152356 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:25.143921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33873 (local address 127.0.97.126:42537)
0927 18:33:25.144147 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:25.144153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:25.144435 (+ 282us) server_negotiation.cc:407] Connection header received
0927 18:33:25.144489 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:25.144493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:25.144553 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:25.144631 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:25.145328 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.146160 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.147421 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.147844 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.147954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.152122 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:33873: BlockingRecv error: recv got EOF from 127.0.97.67:33873 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:25.225721 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:25.217520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:35929)
0927 18:33:25.218116 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:25.218131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:25.218269 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:25.218586 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:25.218590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:25.218978 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:25.219128 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.219136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.220097 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.220100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.221017 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.221028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.221712 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.221717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.221879 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.222485 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:25.222501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:25.225500 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":372,"threads_started":1}
W20260927 18:33:25.225916 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:25.217574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35929 (local address 127.0.97.126:42537)
0927 18:33:25.217732 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:25.217736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:25.218211 (+ 475us) server_negotiation.cc:407] Connection header received
0927 18:33:25.218379 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:25.218383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:25.218454 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:25.218531 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:25.219243 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.219977 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.221137 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.221575 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.221662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.225666 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:35929: BlockingRecv error: recv got EOF from 127.0.97.65:35929 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:25 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:25.689291 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:25.680934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:46251)
0927 18:33:25.681468 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:25.681483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:25.681640 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:25.682013 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:25.682018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:25.682431 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:25.682591 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.682599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.683596 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.683604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.684574 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:25.684585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.685294 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.685300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:25.685483 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.686106 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:25.686146 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:25.689021 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":296,"threads_started":1}
W20260927 18:33:25.689586 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:25.681000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46251 (local address 127.0.97.126:42537)
0927 18:33:25.681233 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:25.681241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:25.681573 (+ 332us) server_negotiation.cc:407] Connection header received
0927 18:33:25.681762 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:25.681767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:25.681881 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:25.681984 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:25.682736 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.683451 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.684715 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:25.685145 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:25.685286 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:25.689290 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:46251: BlockingRecv error: recv got EOF from 127.0.97.66:46251 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:26.168274 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:26.153092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:54641)
0927 18:33:26.153602 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:26.153618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:26.153800 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:26.154160 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:26.154165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:26.154551 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:26.154700 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.154708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.155720 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.155727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.156672 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.156682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.157373 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.157379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.157560 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.165185 (+ 7625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:26.165203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:26.167938 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":315,"threads_started":1}
W20260927 18:33:26.168418 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:26.153144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54641 (local address 127.0.97.126:42537)
0927 18:33:26.153384 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:26.153401 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:26.153712 (+ 311us) server_negotiation.cc:407] Connection header received
0927 18:33:26.153921 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:26.153927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:26.154004 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:26.154113 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:26.154849 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.155580 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.156809 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.157229 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.157362 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.168146 (+ 10784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:54641: BlockingRecv error: recv got EOF from 127.0.97.67:54641 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:26.235349 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:26.226702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:47903)
0927 18:33:26.227178 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:26.227194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:26.227306 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:26.227692 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:26.227696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:26.228109 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:26.228329 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.228342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.229512 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.229524 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.230475 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.230485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.231245 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.231250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.231370 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.232116 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:26.232134 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:26.235006 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
W20260927 18:33:26.235431 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:26.226744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47903 (local address 127.0.97.126:42537)
0927 18:33:26.227000 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:26.227006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:26.227275 (+ 269us) server_negotiation.cc:407] Connection header received
0927 18:33:26.227478 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:26.227483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:26.227559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:26.227648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:26.228521 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.229277 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.230615 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.231051 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.231151 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.235190 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:47903: BlockingRecv error: recv got EOF from 127.0.97.65:47903 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:26 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:26.699075 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:26.690308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:40085)
0927 18:33:26.691005 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:26.691028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:26.691187 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:26.691541 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:26.691546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:26.691980 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:26.692136 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.692145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.693167 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.693173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.694149 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:26.694160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.694928 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.694936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:26.695108 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.695784 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:26.695803 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:26.698675 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
W20260927 18:33:26.699110 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:26.690505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40085 (local address 127.0.97.126:42537)
0927 18:33:26.690665 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:26.690670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:26.691118 (+ 448us) server_negotiation.cc:407] Connection header received
0927 18:33:26.691320 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:26.691325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:26.691403 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:26.691508 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:26.692289 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.693028 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.694295 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:26.694736 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:26.694892 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:26.698882 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:40085: BlockingRecv error: recv got EOF from 127.0.97.66:40085 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:27.178360 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:27.169235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:51363)
0927 18:33:27.169759 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:27.169802 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:27.169916 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:27.170283 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:27.170291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:27.170752 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:27.170916 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.170925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.172078 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.172082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.172995 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.173006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.173999 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.174007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.174186 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.175003 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:27.175022 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:27.178116 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":290,"threads_started":1}
W20260927 18:33:27.178493 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:27.169360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51363 (local address 127.0.97.126:42537)
0927 18:33:27.169528 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:27.169532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:27.169876 (+ 344us) server_negotiation.cc:407] Connection header received
0927 18:33:27.169998 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:27.170002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:27.170061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:27.170148 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:27.171077 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.171910 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.173131 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.173766 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.173938 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.178279 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:51363: BlockingRecv error: recv got EOF from 127.0.97.67:51363 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:27.244563 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:27.236320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:50797)
0927 18:33:27.236991 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:27.237009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:27.237188 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:27.237579 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:27.237583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:27.237968 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:27.238139 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.238147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.239115 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.239119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.240041 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.240052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.240698 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.240703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.240817 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.241411 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:27.241428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:27.244286 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":447,"threads_started":1}
W20260927 18:33:27.244882 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:27.236398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:50797 (local address 127.0.97.126:42537)
0927 18:33:27.236696 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:27.236700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:27.237112 (+ 412us) server_negotiation.cc:407] Connection header received
0927 18:33:27.237313 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:27.237317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:27.237408 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:27.237492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:27.238267 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.238985 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.240159 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.240575 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.240661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.244579 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:50797: BlockingRecv error: recv got EOF from 127.0.97.65:50797 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:27 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:27.708330 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:27.700116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:59507)
0927 18:33:27.700699 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:27.700714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:27.700883 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:27.701237 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:27.701241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:27.701721 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:27.701997 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.702009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.702979 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.702983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.703907 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:27.703918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.704597 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.704602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:27.704725 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.705302 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:27.705318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:27.708077 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":348,"threads_started":1}
W20260927 18:33:27.708444 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:27.700286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:59507 (local address 127.0.97.126:42537)
0927 18:33:27.700552 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:27.700558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:27.700798 (+ 240us) server_negotiation.cc:407] Connection header received
0927 18:33:27.701006 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:27.701012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:27.701077 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:27.701186 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:27.702128 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.702852 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.704039 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:27.704473 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:27.704562 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:27.708250 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:59507: BlockingRecv error: recv got EOF from 127.0.97.66:59507 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:28.187971 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:28.179352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.67:57449)
0927 18:33:28.179885 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:28.179902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:28.180049 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:28.180503 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:28.180506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:28.180838 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:28.181011 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.181019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.182040 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.182044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.182981 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.182992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.183710 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.183714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.183841 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.184575 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:28.184593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:28.187645 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
W20260927 18:33:28.188169 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:28.179435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57449 (local address 127.0.97.126:42537)
0927 18:33:28.179608 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:28.179612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:28.180019 (+ 407us) server_negotiation.cc:407] Connection header received
0927 18:33:28.180270 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:28.180276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:28.180363 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:28.180466 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:28.181159 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.181915 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.183125 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.183584 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.183688 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.187904 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:57449: BlockingRecv error: recv got EOF from 127.0.97.67:57449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:28.255007 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:28.246033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.65:49183)
0927 18:33:28.246755 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:28.246772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:28.246948 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:28.247337 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:28.247342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:28.247783 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:28.247965 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.247973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.248985 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.248991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.249987 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.249998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.250734 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.250739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.250956 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.251636 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:28.251659 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:28.254644 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":393,"threads_started":1}
W20260927 18:33:28.255054 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:28.246277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49183 (local address 127.0.97.126:42537)
0927 18:33:28.246526 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:28.246537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:28.246872 (+ 335us) server_negotiation.cc:407] Connection header received
0927 18:33:28.247085 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:28.247091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:28.247192 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:28.247311 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:28.248138 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.248850 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.250124 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.250557 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.250727 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.254833 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:49183: BlockingRecv error: recv got EOF from 127.0.97.65:49183 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:33:28 dist-test-slave-n3p4 krb5kdc[8605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:33:28.718743 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:28.709613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42537 (local address 127.0.97.66:54947)
0927 18:33:28.710214 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:28.710232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:28.710364 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:28.710701 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:28.710705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:28.711166 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:28.711468 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.711480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.712877 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.712885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.713983 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:28.713996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.714787 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.714792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:28.714938 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.715635 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:28.715653 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:28.718490 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:42537: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":344,"threads_started":1}
W20260927 18:33:28.718981 9341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:28.709691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:54947 (local address 127.0.97.126:42537)
0927 18:33:28.709895 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:28.709900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:28.710376 (+ 476us) server_negotiation.cc:407] Connection header received
0927 18:33:28.710475 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:28.710479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:28.710541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:28.710619 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:28.711645 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.712697 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.714145 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:28.714635 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:28.714740 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:28.718692 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:54947: BlockingRecv error: recv got EOF from 127.0.97.66:54947 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:33:29.022678 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9192
I20260927 18:33:29.035876 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:29.024035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.0.97.65:40735)
0927 18:33:29.024601 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:29.024609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:29.024638 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:33:29.024703 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:29.024710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:29.024955 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:29.025127 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:29.026716 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.027785 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.029207 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.029985 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.030093 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:29.032594 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:29.032604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:29.032619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:29.032657 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:29.035302 (+ 2645us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:33:29.035653 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35796: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
W20260927 18:33:29.035989 9652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35796: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:33:29.036216 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9192: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:40735: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:33:29.036432 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9192
I20260927 18:33:29.053507 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9343
I20260927 18:33:29.066138 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:29.054762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.97.66:38819)
0927 18:33:29.055311 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:29.055318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:29.055348 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:33:29.055406 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:29.055412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:29.055644 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:29.055839 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:29.057134 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.058310 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.059535 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.060104 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.060249 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:29.062892 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:29.062901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:29.062915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:29.062948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:29.065281 (+ 2333us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:33:29.065656 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51736: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":267,"threads_started":1}
W20260927 18:33:29.066192 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9343: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:38819: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:33:29.066249 9653 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51736: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:33:29.066350 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9343
I20260927 18:33:29.082912 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9474
I20260927 18:33:29.095213 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:29.083994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.0.97.67:35121)
0927 18:33:29.084655 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:29.084662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:29.084689 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:33:29.084760 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:29.084767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:29.085010 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:29.085184 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:29.086544 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.087585 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.088767 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.089323 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.089472 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:29.091952 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:29.091960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:29.091975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:29.092012 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:29.094717 (+ 2705us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:33:29.094984 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51774: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
W20260927 18:33:29.095335 9654 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51774: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:33:29.095435 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9474: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:35121: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:33:29.095599 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9474
I20260927 18:33:29.112119 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9123
I20260927 18:33:29.113515 9185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:29.229122 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9123
2026-09-27T18:33:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21115 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:33:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:29Z Disabled control of system clock
I20260927 18:33:29.308895 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33151
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:33:29.650532 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:29.650861 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:29.651000 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:29.658488 9659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:29.658615 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:29.658707 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:29.658804 9659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:29.658880 9659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:29.668383 9659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:29.669924 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:29.671801 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:29.684269 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:29.684176 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:29.684520 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:29.684947 9659 server_base.cc:1034] running on GCE node
I20260927 18:33:29.685760 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:29.687402 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:29.688620 9659 hybrid_clock.cc:648] HybridClock initialized: now 1790534009688582 us; error 33 us; skew 500 ppm
I20260927 18:33:29.689014 9659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:29.691534 9659 webserver.cc:460] Webserver started at http://127.0.97.126:35347/ using document root <none> and password file <none>
I20260927 18:33:29.692276 9659 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:29.692363 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:29.692683 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:29.695654 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "a3aa87c637ca42e7ba99ab406d8e49a7"
format_stamp: "Formatted at 2026-09-27 18:33:29 on dist-test-slave-n3p4"
server_key: "d4be3bdaa402ba12f2e5cb2e9d262641"
server_key_iv: "0aa41c0bfe01939af1849b8e176ac6b1"
server_key_version: "encryptionkey@0"
I20260927 18:33:29.696434 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "a3aa87c637ca42e7ba99ab406d8e49a7"
format_stamp: "Formatted at 2026-09-27 18:33:29 on dist-test-slave-n3p4"
server_key: "d4be3bdaa402ba12f2e5cb2e9d262641"
server_key_iv: "0aa41c0bfe01939af1849b8e176ac6b1"
server_key_version: "encryptionkey@0"
I20260927 18:33:29.701574 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 18:33:29.705188 9673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:29.706822 9659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:33:29.707023 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "a3aa87c637ca42e7ba99ab406d8e49a7"
format_stamp: "Formatted at 2026-09-27 18:33:29 on dist-test-slave-n3p4"
server_key: "d4be3bdaa402ba12f2e5cb2e9d262641"
server_key_iv: "0aa41c0bfe01939af1849b8e176ac6b1"
server_key_version: "encryptionkey@0"
I20260927 18:33:29.707245 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:29.727191 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:29.728103 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:29.728412 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:29.747298 9659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33151
I20260927 18:33:29.747304 9724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33151 every 8 connection(s)
I20260927 18:33:29.748620 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:29.751646 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9659
I20260927 18:33:29.751952 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:33:29.752590 9725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:29.752600 389 external_mini_cluster.cc:1428] Setting key fe9411f08e289038d8cfe104b70c0c6b
I20260927 18:33:29.767764 9725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7: Bootstrap starting.
I20260927 18:33:29.768280 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:29.754672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.0.97.126:33151)
0927 18:33:29.755762 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:29.755781 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:29.755838 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:33:29.757606 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:29.757636 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:29.758277 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:29.758720 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:29.759262 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.761249 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.762661 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:29.763779 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:29.763995 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:29.765317 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:29.765331 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:29.765353 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:29.766121 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:29.766229 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:29.766257 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:29.766376 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:29.766623 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 18:33:29.766980 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":357,"threads_started":1}
I20260927 18:33:29.771135 9725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:29.772226 9725 log.cc:824] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:29.775329 9725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7: No bootstrap required, opened a new log
I20260927 18:33:29.783438 9725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } }
I20260927 18:33:29.783866 9725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:29.783996 9725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3aa87c637ca42e7ba99ab406d8e49a7, State: Initialized, Role: FOLLOWER
I20260927 18:33:29.784720 9725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } }
I20260927 18:33:29.784961 9725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:29.785101 9725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:29.785301 9725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:29.787820 9725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } }
I20260927 18:33:29.788492 9725 leader_election.cc:302] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3aa87c637ca42e7ba99ab406d8e49a7; no voters:
I20260927 18:33:29.789144 9725 leader_election.cc:288] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:33:29.789299 9730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:29.790005 9730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [term 1 LEADER]: Becoming Leader. State: Replica: a3aa87c637ca42e7ba99ab406d8e49a7, State: Running, Role: LEADER
I20260927 18:33:29.790547 9730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } }
I20260927 18:33:29.791393 9725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:29.793429 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3aa87c637ca42e7ba99ab406d8e49a7. Latest consensus state: current_term: 1 leader_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } } }
I20260927 18:33:29.793931 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:29.793754 9731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aa87c637ca42e7ba99ab406d8e49a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33151 } } }
I20260927 18:33:29.794071 9731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:29.795795 9738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:29.800361 9738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:29.808802 9738 catalog_manager.cc:1347] Generated new cluster ID: dad9dba5653c4343b658cf27d61d1a3f
I20260927 18:33:29.808948 9738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:29.820830 9738 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:33:29.822261 9738 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:29.829949 9738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3aa87c637ca42e7ba99ab406d8e49a7: Generated new TSK 0
I20260927 18:33:29.830856 9738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:33:29.839928 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--builtin_ntp_servers=127.0.97.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:33:30.173584 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:30.173928 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:30.174015 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:30.181310 9749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:30.181414 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:30.181516 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:30.190696 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:30.192332 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:30.194137 9749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:30.207424 9754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:30.207536 9755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:30.207573 9757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:30.208629 9749 server_base.cc:1034] running on GCE node
I20260927 18:33:30.209328 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:30.210567 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:30.211808 9749 hybrid_clock.cc:648] HybridClock initialized: now 1790534010211758 us; error 41 us; skew 500 ppm
I20260927 18:33:30.212160 9749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:30.214547 9749 webserver.cc:460] Webserver started at http://127.0.97.65:42263/ using document root <none> and password file <none>
I20260927 18:33:30.215309 9749 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:30.215408 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:30.215756 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:30.218662 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "751338a7500844cea88fd6f7fac8b995"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "14a7ec10403a844f83257fe9a6e73ed5"
server_key_iv: "c1a9c88513b0f4d8a42223584db1207a"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.219445 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "751338a7500844cea88fd6f7fac8b995"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "14a7ec10403a844f83257fe9a6e73ed5"
server_key_iv: "c1a9c88513b0f4d8a42223584db1207a"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.224521 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:33:30.228253 9763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.229755 9749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:30.230012 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "751338a7500844cea88fd6f7fac8b995"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "14a7ec10403a844f83257fe9a6e73ed5"
server_key_iv: "c1a9c88513b0f4d8a42223584db1207a"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.230237 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:30.252929 9749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:30.253914 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:30.254213 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:30.255204 9749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:30.256803 9749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:30.256935 9749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.257072 9749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:30.257164 9749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.290638 9749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40257
I20260927 18:33:30.290730 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40257 every 8 connection(s)
I20260927 18:33:30.292066 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:30.302506 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9749
I20260927 18:33:30.302722 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:33:30.303435 389 external_mini_cluster.cc:1428] Setting key 3e8dc63a6a10ae65a90f55c38ccd14ff
I20260927 18:33:30.305292 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:30.294770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55621 (local address 127.0.97.126:33151)
0927 18:33:30.295365 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:30.295373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:30.296594 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 18:33:30.298410 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:30.298415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:30.298483 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:30.298585 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:30.299613 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.300341 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.302069 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.302758 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.302902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:30.304427 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:30.304434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:30.304438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:30.304458 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:30.304509 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:30.304525 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:30.304619 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:30.304848 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:33:30.304932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":285,"threads_started":1}
I20260927 18:33:30.306448 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--builtin_ntp_servers=127.0.97.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:33:30.306543 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:30.295429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33151 (local address 127.0.97.65:55621)
0927 18:33:30.296436 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:30.296482 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:30.298111 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:30.298785 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:30.298792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:30.298896 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:30.299457 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:30.299473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.300516 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.300519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:30.301745 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:30.301756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.303041 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.303046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:30.303248 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:30.304064 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:30.304068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:30.304122 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:30.304234 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:30.304639 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:30.304649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:30.304656 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:33:30.304810 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 18:33:30.305145 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":324,"threads_started":1}
I20260927 18:33:30.307989 9876 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33151
I20260927 18:33:30.308282 9876 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:30.309006 9876 heartbeater.cc:511] Master 127.0.97.126:33151 requested a full tablet report, sending...
I20260927 18:33:30.310992 9690 ts_manager.cc:194] Registered new tserver with Master: 751338a7500844cea88fd6f7fac8b995 (127.0.97.65:40257)
I20260927 18:33:30.312537 9690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:55621
W20260927 18:33:30.627040 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:30.627355 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:30.627449 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:30.634547 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:30.634646 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:30.634759 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:30.643850 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:30.645069 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:30.646840 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:30.659860 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:30.659839 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:30.660089 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:30.660697 9880 server_base.cc:1034] running on GCE node
I20260927 18:33:30.661285 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:30.662243 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:30.663473 9880 hybrid_clock.cc:648] HybridClock initialized: now 1790534010663429 us; error 55 us; skew 500 ppm
I20260927 18:33:30.663803 9880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:30.666164 9880 webserver.cc:460] Webserver started at http://127.0.97.66:44867/ using document root <none> and password file <none>
I20260927 18:33:30.666841 9880 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:30.666968 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:30.667287 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:30.670143 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "484e1ade308a4ccbb7cd7971cf05b635"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "fe9de80bcee13ea81573ae305c90d9d2"
server_key_iv: "5a22065117839b48b7f43f680b2f1769"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.670917 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "484e1ade308a4ccbb7cd7971cf05b635"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "fe9de80bcee13ea81573ae305c90d9d2"
server_key_iv: "5a22065117839b48b7f43f680b2f1769"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.675817 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:33:30.679186 9894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.680557 9880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:33:30.680730 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "484e1ade308a4ccbb7cd7971cf05b635"
format_stamp: "Formatted at 2026-09-27 18:33:30 on dist-test-slave-n3p4"
server_key: "fe9de80bcee13ea81573ae305c90d9d2"
server_key_iv: "5a22065117839b48b7f43f680b2f1769"
server_key_version: "encryptionkey@0"
I20260927 18:33:30.680944 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:30.718237 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:30.719132 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:30.719416 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:30.720402 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:30.721967 9880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:30.722095 9880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.722220 9880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:30.722313 9880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:30.756815 9880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:42941
I20260927 18:33:30.756850 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:42941 every 8 connection(s)
I20260927 18:33:30.758167 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:30.759574 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 9880
I20260927 18:33:30.759749 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:33:30.760267 389 external_mini_cluster.cc:1428] Setting key d4b7c221e4cb14823f59841a76baf3f8
I20260927 18:33:30.763273 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--builtin_ntp_servers=127.0.97.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:33:30.773437 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:30.761124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52897 (local address 127.0.97.126:33151)
0927 18:33:30.761296 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:30.761302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:30.763212 (+ 1910us) server_negotiation.cc:407] Connection header received
0927 18:33:30.766046 (+ 2834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:30.766051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:30.766123 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:30.766206 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:30.767530 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.768209 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.770706 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.771083 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.771186 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:30.772736 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:30.772741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:30.772744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:30.772762 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:30.772792 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:30.772803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:30.772873 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:30.773114 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:33:30.773187 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:33:30.774883 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:30.761819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33151 (local address 127.0.97.66:52897)
0927 18:33:30.763047 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:30.763095 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:30.765741 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:30.766450 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:30.766463 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:30.766579 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:30.767354 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:30.767379 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.768363 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.768371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:30.770539 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:30.770556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:30.771312 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:30.771321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:30.771629 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:30.772393 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:30.772398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:30.772448 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:30.772549 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:30.772903 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:30.772915 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:30.772926 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:33:30.773101 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 18:33:30.773506 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":459,"threads_started":1}
I20260927 18:33:30.776235 10007 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33151
I20260927 18:33:30.776582 10007 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:30.777308 10007 heartbeater.cc:511] Master 127.0.97.126:33151 requested a full tablet report, sending...
I20260927 18:33:30.778846 9690 ts_manager.cc:194] Registered new tserver with Master: 484e1ade308a4ccbb7cd7971cf05b635 (127.0.97.66:42941)
I20260927 18:33:30.779660 9690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:52897
W20260927 18:33:31.088042 10010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:31.088299 10010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:31.088467 10010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:31.095764 10010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:31.095856 10010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:31.096028 10010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:31.104848 10010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:31.106155 10010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:31.107942 10010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:31.120844 10015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:31.120812 10016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:31.120993 10018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:31.121549 10010 server_base.cc:1034] running on GCE node
I20260927 18:33:31.122189 10010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:31.123083 10010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:31.124296 10010 hybrid_clock.cc:648] HybridClock initialized: now 1790534011124244 us; error 46 us; skew 500 ppm
I20260927 18:33:31.124601 10010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:31.126797 10010 webserver.cc:460] Webserver started at http://127.0.97.67:39069/ using document root <none> and password file <none>
I20260927 18:33:31.127465 10010 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:31.127590 10010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:31.127909 10010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:31.130669 10010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "1e60794b0efd417cbd807ff6dd00a9ef"
format_stamp: "Formatted at 2026-09-27 18:33:31 on dist-test-slave-n3p4"
server_key: "ac3af82390bd5ca77c0f756925e16eeb"
server_key_iv: "da11c2b751c1a415468a512ea81da522"
server_key_version: "encryptionkey@0"
I20260927 18:33:31.131419 10010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "1e60794b0efd417cbd807ff6dd00a9ef"
format_stamp: "Formatted at 2026-09-27 18:33:31 on dist-test-slave-n3p4"
server_key: "ac3af82390bd5ca77c0f756925e16eeb"
server_key_iv: "da11c2b751c1a415468a512ea81da522"
server_key_version: "encryptionkey@0"
I20260927 18:33:31.136555 10010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:33:31.139801 10024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:31.141100 10010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:31.141263 10010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "1e60794b0efd417cbd807ff6dd00a9ef"
format_stamp: "Formatted at 2026-09-27 18:33:31 on dist-test-slave-n3p4"
server_key: "ac3af82390bd5ca77c0f756925e16eeb"
server_key_iv: "da11c2b751c1a415468a512ea81da522"
server_key_version: "encryptionkey@0"
I20260927 18:33:31.141510 10010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:31.189500 10010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:31.190465 10010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:31.190761 10010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:31.191710 10010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:31.193210 10010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:31.193327 10010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:31.193470 10010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:31.193557 10010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:31.227063 10010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:32877
I20260927 18:33:31.227104 10136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:32877 every 8 connection(s)
I20260927 18:33:31.228366 10010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:31.228953 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10010
I20260927 18:33:31.229164 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:33:31.229768 389 external_mini_cluster.cc:1428] Setting key 8610d209ba97768d56255f430fcb44c1
I20260927 18:33:31.241120 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:31.231088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60481 (local address 127.0.97.126:33151)
0927 18:33:31.231281 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:31.231289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:31.232917 (+ 1628us) server_negotiation.cc:407] Connection header received
0927 18:33:31.235019 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:31.235024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:31.235110 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:31.235212 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:31.236231 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.236953 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.238525 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.238920 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.239047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:31.240337 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:31.240341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:31.240345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:31.240375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:31.240402 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:31.240414 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:31.240506 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:31.240715 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:33:31.240805 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:33:31.242159 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:31.231751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33151 (local address 127.0.97.67:60481)
0927 18:33:31.232748 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:31.232796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:31.234768 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:31.235441 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:31.235451 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:31.235534 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:31.236073 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:31.236088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.237091 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.237096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:31.238385 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:31.238397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.239124 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.239131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:31.239352 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:31.240038 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:31.240041 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:31.240077 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:31.240144 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:31.240505 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:31.240516 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:31.240527 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:33:31.240703 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 18:33:31.241066 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":327,"threads_started":1}
I20260927 18:33:31.243523 10137 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33151
I20260927 18:33:31.243904 10137 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:31.244890 10137 heartbeater.cc:511] Master 127.0.97.126:33151 requested a full tablet report, sending...
I20260927 18:33:31.246456 9690 ts_manager.cc:194] Registered new tserver with Master: 1e60794b0efd417cbd807ff6dd00a9ef (127.0.97.67:32877)
I20260927 18:33:31.247182 9690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:60481
I20260927 18:33:31.257941 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:33:31.263933 9878 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:31.262529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.97.126:33151)
0927 18:33:31.262767 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:31.262774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:31.263077 (+ 303us) server_negotiation.cc:407] Connection header received
0927 18:33:31.263203 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:31.263207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:31.263276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:31.263396 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:31.263700 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50140: BlockingRecv error: recv got EOF from 127.0.0.1:50140 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:33:31.264686 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9749
I20260927 18:33:31.272716 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:31.265630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.97.65:40257)
0927 18:33:31.266140 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:31.266147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:31.266171 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:33:31.266223 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:31.266229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:31.266440 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:31.266575 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:31.266973 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.268046 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.269380 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.270091 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.270213 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:31.271237 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:31.271252 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:31.271266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:31.271920 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:31.271969 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:31.271989 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:31.272085 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:31.272243 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:33:31.272406 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":274,"threads_started":1}
I20260927 18:33:31.274014 9871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:31.355003 9876 heartbeater.cc:503] Master 127.0.97.126:33151 was elected leader, sending a full tablet report...
I20260927 18:33:31.356637 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9749
W20260927 18:33:31.374128 9674 connection.cc:535] server connection from 127.0.97.65:55621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:33:31.374306 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9880
I20260927 18:33:31.382583 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:31.375499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.97.66:42941)
0927 18:33:31.376045 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:31.376052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:31.376078 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:33:31.376142 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:31.376148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:31.376386 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:31.376565 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:31.376962 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.378082 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.379275 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.379837 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.379978 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:31.381023 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:31.381031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:31.381045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:31.381662 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:31.381719 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:31.381738 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:31.381936 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:31.382083 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:33:31.382218 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":277,"threads_started":1}
I20260927 18:33:31.383908 10002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:31.466380 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9880
I20260927 18:33:31.482227 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10010
I20260927 18:33:31.490175 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:31.483095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.0.97.67:32877)
0927 18:33:31.483627 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:31.483634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:31.483663 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:33:31.483722 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:31.483729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:31.483961 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:31.484134 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:31.484581 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.485572 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.486914 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:31.487466 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:31.487606 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:31.488775 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:31.488783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:31.488797 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:31.489376 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:31.489422 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:31.489440 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:31.489572 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:31.489684 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:33:31.489834 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
I20260927 18:33:31.491417 10132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:31.573153 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10010
I20260927 18:33:31.589916 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 9659
I20260927 18:33:31.591188 9720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:31.691890 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 9659
2026-09-27T18:33:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2455 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:31Z Disabled control of system clock
I20260927 18:33:31.767511 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35031
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:33:32.095155 10156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:32.095458 10156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:32.095656 10156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:32.102909 10156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:32.103024 10156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:32.103066 10156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:32.103132 10156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:32.103179 10156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:32.112617 10156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:32.113909 10156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:32.115746 10156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:32.128309 10161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:32.128278 10162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:32.128224 10164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:32.129142 10156 server_base.cc:1034] running on GCE node
I20260927 18:33:32.129999 10156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:32.131372 10156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:32.132583 10156 hybrid_clock.cc:648] HybridClock initialized: now 1790534012132533 us; error 38 us; skew 500 ppm
I20260927 18:33:32.132886 10156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:32.135197 10156 webserver.cc:460] Webserver started at http://127.0.97.126:44405/ using document root <none> and password file <none>
I20260927 18:33:32.135819 10156 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:32.135910 10156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:32.136176 10156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:32.139109 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "c605acfbad5245dcacd54366f41d284d"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "a7c1f207acc5e735bb72406b98d5f9ae"
server_key_iv: "5520a4847bf27f731aecfc0c103904b8"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.139930 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "c605acfbad5245dcacd54366f41d284d"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "a7c1f207acc5e735bb72406b98d5f9ae"
server_key_iv: "5520a4847bf27f731aecfc0c103904b8"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.144977 10156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:33:32.148464 10170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:32.149861 10156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:32.149999 10156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "c605acfbad5245dcacd54366f41d284d"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "a7c1f207acc5e735bb72406b98d5f9ae"
server_key_iv: "5520a4847bf27f731aecfc0c103904b8"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.150173 10156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:32.155687 10156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:32.156385 10156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:32.156633 10156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:32.174899 10156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35031
I20260927 18:33:32.174917 10221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35031 every 8 connection(s)
I20260927 18:33:32.176221 10156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:32.177613 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10156
I20260927 18:33:32.177822 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:33:32.178660 389 external_mini_cluster.cc:1428] Setting key 8debd82d86efcd1f91586a41b2ffd384
I20260927 18:33:32.180893 10222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:32.186859 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:32.180678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.0.97.126:35031)
0927 18:33:32.181720 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:32.181730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:32.181769 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:33:32.183308 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:32.183327 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:32.183729 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:32.184059 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:32.185194 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:32.185201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:32.185213 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:32.185228 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:32.185310 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:32.185356 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:32.185535 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:32.185641 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:33:32.185958 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":425,"threads_started":1}
I20260927 18:33:32.192203 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d: Bootstrap starting.
I20260927 18:33:32.195078 10222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:32.196007 10222 log.cc:824] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:32.198817 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d: No bootstrap required, opened a new log
I20260927 18:33:32.205674 10222 raft_consensus.cc:348] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } }
I20260927 18:33:32.206106 10222 raft_consensus.cc:374] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:32.206212 10222 raft_consensus.cc:729] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c605acfbad5245dcacd54366f41d284d, State: Initialized, Role: FOLLOWER
I20260927 18:33:32.206827 10222 consensus_queue.cc:260] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } }
I20260927 18:33:32.207068 10222 raft_consensus.cc:388] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:32.207211 10222 raft_consensus.cc:482] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:32.207396 10222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:32.209564 10222 raft_consensus.cc:504] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } }
I20260927 18:33:32.210140 10222 leader_election.cc:302] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c605acfbad5245dcacd54366f41d284d; no voters:
I20260927 18:33:32.210803 10222 leader_election.cc:288] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:33:32.210980 10227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:32.211743 10227 raft_consensus.cc:686] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [term 1 LEADER]: Becoming Leader. State: Replica: c605acfbad5245dcacd54366f41d284d, State: Running, Role: LEADER
I20260927 18:33:32.212270 10227 consensus_queue.cc:237] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } }
I20260927 18:33:32.212821 10222 sys_catalog.cc:564] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:32.214910 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c605acfbad5245dcacd54366f41d284d. Latest consensus state: current_term: 1 leader_uuid: "c605acfbad5245dcacd54366f41d284d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } } }
I20260927 18:33:32.214964 10228 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c605acfbad5245dcacd54366f41d284d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605acfbad5245dcacd54366f41d284d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35031 } } }
I20260927 18:33:32.215369 10228 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:32.215373 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:32.218225 10233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:32.223312 10233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:32.232668 10233 catalog_manager.cc:1347] Generated new cluster ID: 6412ad6c0a99474bb0f57ec3ae6f9758
I20260927 18:33:32.232782 10233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:32.244617 10233 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:33:32.245019 10233 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:32.255543 10233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c605acfbad5245dcacd54366f41d284d: Generated new TSK 0
I20260927 18:33:32.256325 10233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:33:32.269345 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--builtin_ntp_servers=127.0.97.84:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:33:32.613031 10246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:32.613379 10246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:32.613481 10246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:32.621049 10246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:32.621161 10246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:32.621273 10246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:32.630591 10246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:32.631853 10246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:32.633517 10246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:32.646441 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:32.646466 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:32.646384 10254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:32.647113 10246 server_base.cc:1034] running on GCE node
I20260927 18:33:32.647707 10246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:32.648620 10246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:32.649853 10246 hybrid_clock.cc:648] HybridClock initialized: now 1790534012649806 us; error 36 us; skew 500 ppm
I20260927 18:33:32.650185 10246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:32.652369 10246 webserver.cc:460] Webserver started at http://127.0.97.65:40471/ using document root <none> and password file <none>
I20260927 18:33:32.653103 10246 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:32.653218 10246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:32.653581 10246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:32.658150 10246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "302061eec8b64f868b8bcba10d858ce8"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "ef8055f1f014e73b978dbf9926147e16"
server_key_iv: "e86267e2cd953281a17b1d26759835b2"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.658901 10246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "302061eec8b64f868b8bcba10d858ce8"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "ef8055f1f014e73b978dbf9926147e16"
server_key_iv: "e86267e2cd953281a17b1d26759835b2"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.664225 10246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 18:33:32.667721 10260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:32.669207 10246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:32.669415 10246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "302061eec8b64f868b8bcba10d858ce8"
format_stamp: "Formatted at 2026-09-27 18:33:32 on dist-test-slave-n3p4"
server_key: "ef8055f1f014e73b978dbf9926147e16"
server_key_iv: "e86267e2cd953281a17b1d26759835b2"
server_key_version: "encryptionkey@0"
I20260927 18:33:32.669638 10246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:32.675431 10246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:32.676280 10246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:32.676571 10246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:32.677492 10246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:32.678884 10246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:32.678968 10246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:32.679122 10246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:32.679217 10246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:32.712282 10246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34979
I20260927 18:33:32.712407 10372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34979 every 8 connection(s)
I20260927 18:33:32.713768 10246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:32.717962 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10246
I20260927 18:33:32.718146 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:33:32.718976 389 external_mini_cluster.cc:1428] Setting key c5aa7fdbda3ecd11bda795b30c3e543c
I20260927 18:33:32.721923 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--builtin_ntp_servers=127.0.97.84:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:33:32.724323 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:32.717080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35031 (local address 127.0.97.65:42309)
0927 18:33:32.718358 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:32.718409 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:32.720245 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:32.720941 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:32.720950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:32.721044 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:32.721949 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:32.721952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:32.721991 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:32.722108 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:32.722507 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:32.722515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:32.722520 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:33:32.722801 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 18:33:32.723078 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":401,"threads_started":1}
I20260927 18:33:32.724411 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:32.716460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42309 (local address 127.0.97.126:35031)
0927 18:33:32.716976 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:32.716980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:32.718534 (+ 1554us) server_negotiation.cc:407] Connection header received
0927 18:33:32.720501 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:32.720506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:32.720570 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:32.720697 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:32.722276 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:32.722284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:32.722293 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:32.722334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:32.722387 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:32.722405 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:32.722527 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:32.722738 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:33:32.722810 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
I20260927 18:33:32.725935 10373 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35031
I20260927 18:33:32.726259 10373 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:32.727089 10373 heartbeater.cc:511] Master 127.0.97.126:35031 requested a full tablet report, sending...
I20260927 18:33:32.729449 10187 ts_manager.cc:194] Registered new tserver with Master: 302061eec8b64f868b8bcba10d858ce8 (127.0.97.65:34979)
W20260927 18:33:33.054929 10377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:33.055194 10377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:33.055284 10377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:33.062548 10377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:33.062651 10377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:33.062757 10377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:33.071664 10377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:33.072937 10377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:33.074983 10377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:33.088140 10382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:33.088243 10383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:33.088202 10385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:33.089190 10377 server_base.cc:1034] running on GCE node
I20260927 18:33:33.089819 10377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:33.090749 10377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:33.091981 10377 hybrid_clock.cc:648] HybridClock initialized: now 1790534013091949 us; error 48 us; skew 500 ppm
I20260927 18:33:33.092393 10377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:33.094631 10377 webserver.cc:460] Webserver started at http://127.0.97.66:42423/ using document root <none> and password file <none>
I20260927 18:33:33.095336 10377 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:33.095465 10377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:33.095793 10377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:33.098644 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "032fdf5fcdbc4b26b6339aea5f101208"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "456a760bc84446e546391d0c79401a40"
server_key_iv: "9539bd370597f34a75d9b5965284812c"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.099419 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "032fdf5fcdbc4b26b6339aea5f101208"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "456a760bc84446e546391d0c79401a40"
server_key_iv: "9539bd370597f34a75d9b5965284812c"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.104514 10377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:33:33.107949 10391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.109186 10377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:33:33.109366 10377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "032fdf5fcdbc4b26b6339aea5f101208"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "456a760bc84446e546391d0c79401a40"
server_key_iv: "9539bd370597f34a75d9b5965284812c"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.109597 10377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:33.115557 10377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:33.116312 10377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:33.116580 10377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:33.117465 10377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:33.118885 10377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:33.119002 10377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.119136 10377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:33.119227 10377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.152055 10377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37591
I20260927 18:33:33.152108 10503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37591 every 8 connection(s)
I20260927 18:33:33.153479 10377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:33.156445 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10377
I20260927 18:33:33.156631 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:33:33.157073 389 external_mini_cluster.cc:1428] Setting key 6f405c21e26e6ccf6c133726536a306a
I20260927 18:33:33.160109 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--builtin_ntp_servers=127.0.97.84:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:33:33.162638 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.156191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39701 (local address 127.0.97.126:35031)
0927 18:33:33.156472 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.156480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.158191 (+ 1711us) server_negotiation.cc:407] Connection header received
0927 18:33:33.160284 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.160292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.160365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.160466 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.161990 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:33.161994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:33.161997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:33.162011 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:33.162041 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:33.162053 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:33.162141 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:33.162396 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:33:33.162442 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:33:33.163851 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.156846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35031 (local address 127.0.97.66:39701)
0927 18:33:33.158009 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:33.158055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:33.159954 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:33.160718 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:33.160732 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:33.160863 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:33.161618 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:33.161624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:33.161677 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:33.161823 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:33.162152 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:33.162165 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:33.162176 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:33:33.162451 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 18:33:33.162696 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":332,"threads_started":1}
I20260927 18:33:33.165133 10504 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35031
I20260927 18:33:33.165449 10504 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:33.166038 10504 heartbeater.cc:511] Master 127.0.97.126:35031 requested a full tablet report, sending...
I20260927 18:33:33.167563 10187 ts_manager.cc:194] Registered new tserver with Master: 032fdf5fcdbc4b26b6339aea5f101208 (127.0.97.66:37591)
W20260927 18:33:33.486655 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:33.486974 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:33.487130 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:33.494632 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:33.494771 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:33.494954 10507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:33.504086 10507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:33.505393 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:33.507256 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:33.520704 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:33.520745 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:33.520983 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:33.521669 10507 server_base.cc:1034] running on GCE node
I20260927 18:33:33.522382 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:33.523432 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:33.524665 10507 hybrid_clock.cc:648] HybridClock initialized: now 1790534013524633 us; error 85 us; skew 500 ppm
I20260927 18:33:33.524992 10507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:33:33.527240 10507 webserver.cc:460] Webserver started at http://127.0.97.67:38127/ using document root <none> and password file <none>
I20260927 18:33:33.527921 10507 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:33.528050 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:33.528368 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:33.531214 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "31091ab5a9bf49db812771f1cc8047ab"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "316c1a4f30a2eded62feb362f7e936b7"
server_key_iv: "465fb4e542728e851c7946a71d9a929a"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.532012 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "31091ab5a9bf49db812771f1cc8047ab"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "316c1a4f30a2eded62feb362f7e936b7"
server_key_iv: "465fb4e542728e851c7946a71d9a929a"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.537143 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:33:33.540575 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.542032 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:33:33.542202 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "31091ab5a9bf49db812771f1cc8047ab"
format_stamp: "Formatted at 2026-09-27 18:33:33 on dist-test-slave-n3p4"
server_key: "316c1a4f30a2eded62feb362f7e936b7"
server_key_iv: "465fb4e542728e851c7946a71d9a929a"
server_key_version: "encryptionkey@0"
I20260927 18:33:33.542461 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:33.548177 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:33.548988 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:33.549256 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:33.550252 10507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:33.551621 10507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:33.551743 10507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.551870 10507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:33.551993 10507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:33.584611 10507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33829
I20260927 18:33:33.584702 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33829 every 8 connection(s)
I20260927 18:33:33.586004 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:33.594362 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10507
I20260927 18:33:33.594538 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:33:33.594538 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.588823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36587 (local address 127.0.97.126:35031)
0927 18:33:33.589001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.589006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.590502 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 18:33:33.592180 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.592184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.592238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.592332 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.593741 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:33.593745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:33.593748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:33.593764 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:33.593853 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:33.593867 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:33.593975 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:33.594191 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 18:33:33.594243 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:33:33.595080 389 external_mini_cluster.cc:1428] Setting key 1b4630651a88c7c748d49948ddc31c9d
I20260927 18:33:33.595759 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.589266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35031 (local address 127.0.97.67:36587)
0927 18:33:33.590308 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:33.590372 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:33.591890 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:33.592565 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:33.592574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:33.592670 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:33.593432 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:33:33.593436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:33.593479 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:33:33.593567 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:33.593968 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:33.593976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:33.593983 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:33:33.594276 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 18:33:33.594563 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":314,"threads_started":1}
I20260927 18:33:33.597186 10634 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35031
I20260927 18:33:33.597492 10634 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:33.598218 10634 heartbeater.cc:511] Master 127.0.97.126:35031 requested a full tablet report, sending...
I20260927 18:33:33.599452 10187 ts_manager.cc:194] Registered new tserver with Master: 31091ab5a9bf49db812771f1cc8047ab (127.0.97.67:33829)
I20260927 18:33:33.610147 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:33:33.616446 10375 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:33:33.614885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.97.126:35031)
0927 18:33:33.615117 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.615126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.615567 (+ 441us) server_negotiation.cc:407] Connection header received
0927 18:33:33.615682 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.615685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.615732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.615808 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.616193 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56536: BlockingRecv error: recv got EOF from 127.0.0.1:56536 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:33:33.617218 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10246
I20260927 18:33:33.620482 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.618114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.97.65:34979)
0927 18:33:33.618573 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.618582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.618617 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:33:33.618671 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.618677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.618886 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.619005 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.619869 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:33.619876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:33.619887 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:33.619906 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:33.619959 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:33.619982 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:33.620084 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:33.620181 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 18:33:33.620262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260927 18:33:33.621724 10368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:33.703235 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10246
I20260927 18:33:33.719482 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10377
I20260927 18:33:33.723132 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.720395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.97.66:37591)
0927 18:33:33.720883 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.720890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.720915 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:33:33.720975 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.720982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.721203 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.721377 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.722438 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:33.722444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:33.722455 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:33.722467 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:33.722505 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:33.722518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:33.722659 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:33.722762 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:33:33.722862 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260927 18:33:33.724466 10499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:33.804755 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10377
I20260927 18:33:33.820763 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10507
I20260927 18:33:33.825069 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:33.821895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.0.97.67:33829)
0927 18:33:33.822692 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:33.822699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:33.822730 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:33:33.822820 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:33.822829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:33.823126 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:33.823262 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:33.824312 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:33.824318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:33.824329 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:33:33.824344 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:33.824385 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:33:33.824398 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:33.824530 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:33.824628 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:33:33.824739 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":402,"threads_started":1}
I20260927 18:33:33.826521 10629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:33.906921 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10507
I20260927 18:33:33.923017 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10156
I20260927 18:33:33.924114 10217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:34.026577 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10156
2026-09-27T18:33:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2334 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:33:34 dist-test-slave-n3p4 krb5kdc[10652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:33:34 dist-test-slave-n3p4 krb5kdc[10652](info): set up 2 sockets
Sep 27 18:33:34 dist-test-slave-n3p4 krb5kdc[10652](info): commencing operation
krb5kdc: starting...
W20260927 18:33:35.119007 389 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:33:35 dist-test-slave-n3p4 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:33:35.352278 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37519
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:35.681206 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:35.681505 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:35.681658 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:35.689033 10665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:33:35.689136 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:35.689221 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:35.689303 10665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:33:35.689400 10665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:33:35.699568 10665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:35.700889 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:35.702278 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:35.714074 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:35.714105 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:35.714082 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:35.714908 10665 server_base.cc:1034] running on GCE node
I20260927 18:33:35.715715 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:35.717132 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:35.718339 10665 hybrid_clock.cc:648] HybridClock initialized: now 1790534015718269 us; error 54 us; skew 500 ppm
I20260927 18:33:35.718717 10665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:35 dist-test-slave-n3p4 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:35.721933 10665 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:33:35.723443 10665 webserver.cc:460] Webserver started at http://127.0.97.126:40347/ using document root <none> and password file <none>
I20260927 18:33:35.724125 10665 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:35.724224 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:35.724488 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:35.727546 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "d5984480ece04c129d2a473960d1727f"
format_stamp: "Formatted at 2026-09-27 18:33:35 on dist-test-slave-n3p4"
server_key: "7325d6d2c6315296aa49c17ae5a365a4"
server_key_iv: "7ef12a834f33240eb8c60709d0381f23"
server_key_version: "encryptionkey@0"
I20260927 18:33:35.728252 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "d5984480ece04c129d2a473960d1727f"
format_stamp: "Formatted at 2026-09-27 18:33:35 on dist-test-slave-n3p4"
server_key: "7325d6d2c6315296aa49c17ae5a365a4"
server_key_iv: "7ef12a834f33240eb8c60709d0381f23"
server_key_version: "encryptionkey@0"
I20260927 18:33:35.733882 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:33:35.737491 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:35.738989 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:33:35.739204 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "d5984480ece04c129d2a473960d1727f"
format_stamp: "Formatted at 2026-09-27 18:33:35 on dist-test-slave-n3p4"
server_key: "7325d6d2c6315296aa49c17ae5a365a4"
server_key_iv: "7ef12a834f33240eb8c60709d0381f23"
server_key_version: "encryptionkey@0"
I20260927 18:33:35.739411 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:35.760217 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:35.764407 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:35.764695 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:35.783440 10665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37519
I20260927 18:33:35.783478 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37519 every 8 connection(s)
I20260927 18:33:35.784796 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:33:35.787176 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10665
I20260927 18:33:35.787400 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:33:35.788033 389 external_mini_cluster.cc:1428] Setting key 590ffcf8ec1b78bc8063eb50cf894f8e
I20260927 18:33:35.788668 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:35.803289 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: Bootstrap starting.
Sep 27 18:33:35 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:35.807066 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:35.808218 10732 log.cc:824] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:35.811628 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: No bootstrap required, opened a new log
I20260927 18:33:35.812112 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:35.790030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.0.97.126:37519)
0927 18:33:35.790796 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:35.790808 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:35.790847 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:33:35.793103 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:35.793125 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:35.793559 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:35.794042 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:35.795752 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:35.797074 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:35.798859 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:35.799687 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:35.799843 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:35.804649 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:35.804657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:35.804671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:35.804713 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:35.807852 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:35.808676 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:35.808682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:35.808688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:35.808946 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:35.809378 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:35.809384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:35.809388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:35.810014 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:35.810245 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:35.810693 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 18:33:35.811028 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":375,"threads_started":1}
I20260927 18:33:35.819476 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } }
I20260927 18:33:35.819885 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:35.819995 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5984480ece04c129d2a473960d1727f, State: Initialized, Role: FOLLOWER
I20260927 18:33:35.820637 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } }
I20260927 18:33:35.820878 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:33:35.821022 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:33:35.821199 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:35.823411 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } }
I20260927 18:33:35.823972 10732 leader_election.cc:302] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5984480ece04c129d2a473960d1727f; no voters:
I20260927 18:33:35.824707 10732 leader_election.cc:288] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:33:35.824856 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:35.825596 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [term 1 LEADER]: Becoming Leader. State: Replica: d5984480ece04c129d2a473960d1727f, State: Running, Role: LEADER
I20260927 18:33:35.826259 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } }
I20260927 18:33:35.826926 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:33:35.829118 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5984480ece04c129d2a473960d1727f. Latest consensus state: current_term: 1 leader_uuid: "d5984480ece04c129d2a473960d1727f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } } }
I20260927 18:33:35.829144 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5984480ece04c129d2a473960d1727f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5984480ece04c129d2a473960d1727f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37519 } } }
I20260927 18:33:35.829524 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:35.829524 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f [sys.catalog]: This master's current role is: LEADER
I20260927 18:33:35.831967 10744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:33:35.838613 10744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:33:35.848738 10744 catalog_manager.cc:1347] Generated new cluster ID: 0eff67f3a6724ffc8a22de07cc25bc29
I20260927 18:33:35.848971 10744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:33:35.869625 10744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:33:35.870993 10744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:33:35.887617 10744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: Generated new TSK 0
I20260927 18:33:35.888487 10744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:33:35.949174 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:36.289541 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:36.289911 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:36.290104 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:36.297405 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:36.297569 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:36.297809 10758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:33:36.307843 10758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:36.309288 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:36.310709 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:36.324934 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:36.324996 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:36.325021 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:36.326508 10758 server_base.cc:1034] running on GCE node
I20260927 18:33:36.327191 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:36.328212 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:36.329456 10758 hybrid_clock.cc:648] HybridClock initialized: now 1790534016329403 us; error 48 us; skew 500 ppm
I20260927 18:33:36.329766 10758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:36 dist-test-slave-n3p4 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:36.336795 10758 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:33:36.338411 10758 webserver.cc:460] Webserver started at http://127.0.97.65:40115/ using document root <none> and password file <none>
I20260927 18:33:36.339166 10758 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:36.339294 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:36.339555 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:36.342381 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "93627729a0314b18a8e7fada44827f81"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "c20c87d4fd3b698a81a93a63b35bd383"
server_key_iv: "aa8e6a257e4a431452fbcbac46956a0c"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.343161 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "93627729a0314b18a8e7fada44827f81"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "c20c87d4fd3b698a81a93a63b35bd383"
server_key_iv: "aa8e6a257e4a431452fbcbac46956a0c"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.348208 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:33:36.351532 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.352881 10758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:33:36.353035 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "93627729a0314b18a8e7fada44827f81"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "c20c87d4fd3b698a81a93a63b35bd383"
server_key_iv: "aa8e6a257e4a431452fbcbac46956a0c"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.353238 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:36.385671 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:36.390271 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:36.390549 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:36.391479 10758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:36.392736 10758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:36.392807 10758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.392978 10758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:36.393049 10758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.425710 10758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:46433
I20260927 18:33:36.425804 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:46433 every 8 connection(s)
I20260927 18:33:36.427136 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:33:36.436434 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10758
I20260927 18:33:36.436661 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:33:36.437347 389 external_mini_cluster.cc:1428] Setting key e826adfed71143a0ab8310499971f9a9
Sep 27 18:33:36 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:36.447942 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:36.429654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42927 (local address 127.0.97.126:37519)
0927 18:33:36.430123 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:36.430128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:36.431145 (+ 1017us) server_negotiation.cc:407] Connection header received
0927 18:33:36.433027 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:36.433033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:36.433128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:36.433247 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:36.434884 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.435607 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.437415 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.437952 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.438102 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:36.441812 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:36.441823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:36.441829 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:36.441871 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:36.444852 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:36.445606 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:36.445615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:36.445619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:36.445752 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:36.446237 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:36.446245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:36.446248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:36.446534 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:36.446679 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:36.447508 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 18:33:36.447687 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":265,"threads_started":1}
I20260927 18:33:36.449486 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:36.430248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37519 (local address 127.0.97.65:42927)
0927 18:33:36.430948 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:36.431003 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:36.432730 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:36.433506 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:36.433516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:36.434108 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:36.434706 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:36.434724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.435785 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.435798 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:36.437249 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:36.437262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.438199 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.438206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:36.438454 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:36.439207 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:36.439231 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:36.441537 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:36.445006 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:36.445016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:36.445042 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:36.445477 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:36.445897 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:36.445903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:36.445907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:36.446123 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:36.446670 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:36.446678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:36.447037 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 18:33:36.447351 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 18:33:36.447965 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":292,"threads_started":1}
I20260927 18:33:36.451197 10886 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37519
I20260927 18:33:36.451686 10886 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:36.452436 10886 heartbeater.cc:511] Master 127.0.97.126:37519 requested a full tablet report, sending...
I20260927 18:33:36.454648 10697 ts_manager.cc:194] Registered new tserver with Master: 93627729a0314b18a8e7fada44827f81 (127.0.97.65:46433)
I20260927 18:33:36.456106 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:42927
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:33:36.490003 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:36.818380 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:36.818661 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:36.818756 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:36.826217 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:36.826336 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:36.826498 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:33:36.836470 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:36.837791 10892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:36.839089 10892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:36.851877 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:36.851922 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:36.851886 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:36.853026 10892 server_base.cc:1034] running on GCE node
I20260927 18:33:36.853585 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:36.854586 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:36.855818 10892 hybrid_clock.cc:648] HybridClock initialized: now 1790534016855776 us; error 45 us; skew 500 ppm
I20260927 18:33:36.856142 10892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:36 dist-test-slave-n3p4 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:36.859413 10892 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:33:36.860913 10892 webserver.cc:460] Webserver started at http://127.0.97.66:36561/ using document root <none> and password file <none>
I20260927 18:33:36.861580 10892 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:36.861677 10892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:36.862027 10892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:36.864835 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "2bda6a6659bd7272ff16458a8bcf2404"
server_key_iv: "c07da57f287b4920e946591626a83eb9"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.865536 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "2bda6a6659bd7272ff16458a8bcf2404"
server_key_iv: "c07da57f287b4920e946591626a83eb9"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.870868 10892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 18:33:36.874435 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.875962 10892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:33:36.876103 10892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c"
format_stamp: "Formatted at 2026-09-27 18:33:36 on dist-test-slave-n3p4"
server_key: "2bda6a6659bd7272ff16458a8bcf2404"
server_key_iv: "c07da57f287b4920e946591626a83eb9"
server_key_version: "encryptionkey@0"
I20260927 18:33:36.876271 10892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:36.887451 10892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:36.891702 10892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:36.892113 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:36.893010 10892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:36.894339 10892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:36.894411 10892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.894493 10892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:36.894536 10892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:36.927842 10892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:46637
I20260927 18:33:36.927977 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:46637 every 8 connection(s)
I20260927 18:33:36.929126 10892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:33:36.933033 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 10892
I20260927 18:33:36.933219 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:33:36.933749 389 external_mini_cluster.cc:1428] Setting key 01f0404c73975858d53c6fa0a1e50e2e
Sep 27 18:33:36 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:36.950423 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:36.931668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33627 (local address 127.0.97.126:37519)
0927 18:33:36.931891 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:36.931896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:36.933207 (+ 1311us) server_negotiation.cc:407] Connection header received
0927 18:33:36.935613 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:36.935622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:36.935710 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:36.935801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:36.937418 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.938603 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.940306 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.941062 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.941202 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:36.945014 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:36.945019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:36.945022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:36.945053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:36.947559 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:36.948252 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:36.948259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:36.948262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:36.948381 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:36.948774 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:36.948781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:36.948787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:36.949039 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:36.949195 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:36.949847 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 18:33:36.950071 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:33:36.951427 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:36.932129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37519 (local address 127.0.97.66:33627)
0927 18:33:36.933014 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:36.933074 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:36.935269 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:36.936097 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:36.936106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:36.936643 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:36.937234 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:36.937251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.938792 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.938796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:36.940161 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:36.940172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:36.941265 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:36.941269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:36.941489 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:36.942226 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:36.942247 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:36.944782 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:36.947692 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:36.947701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:36.947719 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:36.948136 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:36.948493 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:36.948498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:36.948502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:36.948670 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:36.949159 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:36.949169 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:36.949493 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 18:33:36.949753 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 18:33:36.950285 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":305,"threads_started":1}
I20260927 18:33:36.953028 11020 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37519
I20260927 18:33:36.953357 11020 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:36.954147 11020 heartbeater.cc:511] Master 127.0.97.126:37519 requested a full tablet report, sending...
I20260927 18:33:36.955667 10697 ts_manager.cc:194] Registered new tserver with Master: cbcb280c6a89438fa9e4fd9f5cdb949c (127.0.97.66:46637)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:33:36.956907 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:33:36.989553 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:33:37.307747 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:33:37.308025 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:33:37.308193 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:33:37.315310 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:33:37.315409 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:33:37.315651 11025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:33:37.325337 11025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:33:37.326674 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:33:37.328076 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:33:37.341475 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:37.341444 11033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:33:37.341416 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:33:37.341981 11025 server_base.cc:1034] running on GCE node
I20260927 18:33:37.342701 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:33:37.343675 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:33:37.344903 11025 hybrid_clock.cc:648] HybridClock initialized: now 1790534017344846 us; error 50 us; skew 500 ppm
I20260927 18:33:37.345232 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:33:37 dist-test-slave-n3p4 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:33:37.348495 11025 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:33:37.349982 11025 webserver.cc:460] Webserver started at http://127.0.97.67:39837/ using document root <none> and password file <none>
I20260927 18:33:37.350626 11025 fs_manager.cc:359] Metadata directory not provided
I20260927 18:33:37.350764 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:33:37.351054 11025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:33:37.354014 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "7bde1788dbef43f98c449015d5561561"
format_stamp: "Formatted at 2026-09-27 18:33:37 on dist-test-slave-n3p4"
server_key: "d0bdd3e8db28dde5dd781f27027938c1"
server_key_iv: "930b8cd5255adf9bf0c5402cf6a0052b"
server_key_version: "encryptionkey@0"
I20260927 18:33:37.354741 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "7bde1788dbef43f98c449015d5561561"
format_stamp: "Formatted at 2026-09-27 18:33:37 on dist-test-slave-n3p4"
server_key: "d0bdd3e8db28dde5dd781f27027938c1"
server_key_iv: "930b8cd5255adf9bf0c5402cf6a0052b"
server_key_version: "encryptionkey@0"
I20260927 18:33:37.361007 11025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260927 18:33:37.364497 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:37.366015 11025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:33:37.366214 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "7bde1788dbef43f98c449015d5561561"
format_stamp: "Formatted at 2026-09-27 18:33:37 on dist-test-slave-n3p4"
server_key: "d0bdd3e8db28dde5dd781f27027938c1"
server_key_iv: "930b8cd5255adf9bf0c5402cf6a0052b"
server_key_version: "encryptionkey@0"
I20260927 18:33:37.366420 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:33:37.387409 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:33:37.392412 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:33:37.392726 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:33:37.393704 11025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:33:37.395130 11025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:33:37.395248 11025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:37.395404 11025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:33:37.395481 11025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:33:37.431156 11025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35097
I20260927 18:33:37.431216 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35097 every 8 connection(s)
I20260927 18:33:37.432440 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:33:37.441213 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 11025
I20260927 18:33:37.441396 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:33:37.441932 389 external_mini_cluster.cc:1428] Setting key fa97f9c2f102f7cff752350d285312eb
Sep 27 18:33:37 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:33:37.453063 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.435062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44589 (local address 127.0.97.126:37519)
0927 18:33:37.435273 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.435280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.436516 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 18:33:37.438351 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.438355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.438425 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.438518 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:37.439959 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.440686 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.442400 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.442942 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.443073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.447052 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:37.447059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:37.447065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:37.447108 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:37.449716 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.450552 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.450559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.450563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.450711 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.451190 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.451196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.451199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.451492 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:37.451666 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.452421 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.452641 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:33:37.454089 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.435611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37519 (local address 127.0.97.67:44589)
0927 18:33:37.436314 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.436359 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.438060 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.438716 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.438724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.439271 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:33:37.439817 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.439833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.440821 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.440824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.442250 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.442261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.443195 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.443204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.443449 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.444206 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:33:37.444246 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:33:37.446836 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:33:37.449932 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:37.449942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:37.449961 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:37.450414 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:37.450850 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:33:37.450855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:33:37.450860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:33:37.451060 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:33:37.451639 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:33:37.451648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:33:37.452030 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.452319 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.452863 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
I20260927 18:33:37.455552 11153 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37519
I20260927 18:33:37.455879 11153 heartbeater.cc:464] Registering TS with master...
I20260927 18:33:37.456809 11153 heartbeater.cc:511] Master 127.0.97.126:37519 requested a full tablet report, sending...
I20260927 18:33:37.458469 10697 ts_manager.cc:194] Registered new tserver with Master: 7bde1788dbef43f98c449015d5561561 (127.0.97.67:35097)
I20260927 18:33:37.459277 10886 heartbeater.cc:503] Master 127.0.97.126:37519 was elected leader, sending a full tablet report...
I20260927 18:33:37.459316 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:44589
I20260927 18:33:37.469522 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:33:37.486207 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.473954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.126:37519)
0927 18:33:37.474132 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.474140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.474566 (+ 426us) server_negotiation.cc:407] Connection header received
0927 18:33:37.474748 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.474753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.474823 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.474921 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:37.476211 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.476908 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.478216 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.478626 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.478760 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.481123 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:37.481129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:37.481134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:37.481175 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:37.483646 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.484266 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.484273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.484277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.484410 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.484783 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.484789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.484792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.485098 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:37.485335 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.485663 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.485903 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:33:37.489756 10697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:33:37.497123 10697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:33:37.533217 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.508724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:46637 (local address 127.0.0.1:34534)
0927 18:33:37.522892 (+ 14168us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.522930 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.523114 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.525139 (+ 2025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.525145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.525182 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.525568 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.525583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.527406 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.527413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.529630 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.529644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.532133 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.532139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.532449 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.532519 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.532695 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.532890 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13920,"thread_start_us":16844,"threads_started":1}
I20260927 18:33:37.533214 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.508731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.97.66:46637)
0927 18:33:37.524328 (+ 15597us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.524341 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.524380 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:33:37.524518 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.524529 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.524862 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.525127 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.525833 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.527245 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.530536 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.531955 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.532073 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.532274 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.532733 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.532897 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":17258,"threads_started":1}
I20260927 18:33:37.535378 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.509645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.97.67:35097)
0927 18:33:37.522062 (+ 12417us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.522078 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.525962 (+ 3884us) server_negotiation.cc:407] Connection header received
0927 18:33:37.526571 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.526580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.526869 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.527072 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.528605 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.530132 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.532515 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.533955 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.534051 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.534292 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.534849 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.535033 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12154,"thread_start_us":12461,"threads_started":1}
I20260927 18:33:37.535897 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.509373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:35097 (local address 127.0.0.1:39130)
0927 18:33:37.525851 (+ 16478us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.525869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.526280 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.527720 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.527727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.527769 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.528375 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.528390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.530288 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.530293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.532321 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.532337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.534470 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.534476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.534643 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.534673 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.535394 (+ 721us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.535519 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16338,"mutex_wait_us":4,"thread_start_us":13134,"threads_started":1}
I20260927 18:33:37.537510 10955 tablet_service.cc:1467] Processing CreateTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 (DEFAULT_TABLE table=test-table [id=73ff53e53f8047e59a50b0293d774244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:37.539137 10955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da96fdbc9d54a418d4cf5ee129597c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:37.539561 11088 tablet_service.cc:1467] Processing CreateTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 (DEFAULT_TABLE table=test-table [id=73ff53e53f8047e59a50b0293d774244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:37.540107 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.528724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46433 (local address 127.0.0.1:37942)
0927 18:33:37.532744 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.532761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.532865 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.533934 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.533939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.533964 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.534301 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.534315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.536197 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.536202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.537669 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.537680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.539565 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.539571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.539701 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.539722 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.539804 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.539902 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3886,"thread_start_us":3719,"threads_started":1}
I20260927 18:33:37.540891 11088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da96fdbc9d54a418d4cf5ee129597c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:37.540999 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.526471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.0.97.65:46433)
0927 18:33:37.532155 (+ 5684us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.532163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.532987 (+ 824us) server_negotiation.cc:407] Connection header received
0927 18:33:37.533076 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.533084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.533305 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.534503 (+ 1198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.534725 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.536051 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.537977 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.539353 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.540436 (+ 1083us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.540586 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.540634 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.540740 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5515,"thread_start_us":5246,"threads_started":1}
I20260927 18:33:37.544134 10821 tablet_service.cc:1467] Processing CreateTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 (DEFAULT_TABLE table=test-table [id=73ff53e53f8047e59a50b0293d774244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:33:37.545409 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da96fdbc9d54a418d4cf5ee129597c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:33:37.554267 11171 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Bootstrap starting.
I20260927 18:33:37.555123 11172 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Bootstrap starting.
I20260927 18:33:37.558352 11172 tablet_bootstrap.cc:654] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:37.558543 11171 tablet_bootstrap.cc:654] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:37.559428 11172 log.cc:824] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:37.560146 11171 log.cc:824] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:37.560730 11173 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Bootstrap starting.
I20260927 18:33:37.562657 11172 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: No bootstrap required, opened a new log
I20260927 18:33:37.562949 11172 ts_tablet_manager.cc:1397] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:33:37.563679 11171 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: No bootstrap required, opened a new log
I20260927 18:33:37.563946 11171 ts_tablet_manager.cc:1397] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:33:37.569329 11173 tablet_bootstrap.cc:654] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Neither blocks nor log segments found. Creating new log.
I20260927 18:33:37.570222 11172 raft_consensus.cc:348] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.570644 11172 raft_consensus.cc:374] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:37.570745 11172 raft_consensus.cc:729] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bde1788dbef43f98c449015d5561561, State: Initialized, Role: FOLLOWER
I20260927 18:33:37.570727 11173 log.cc:824] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Log is configured to *not* fsync() on all Append() calls
I20260927 18:33:37.571456 11172 consensus_queue.cc:260] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.574558 11153 heartbeater.cc:503] Master 127.0.97.126:37519 was elected leader, sending a full tablet report...
I20260927 18:33:37.574373 11171 raft_consensus.cc:348] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.574827 11171 raft_consensus.cc:374] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:37.574947 11171 raft_consensus.cc:729] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcb280c6a89438fa9e4fd9f5cdb949c, State: Initialized, Role: FOLLOWER
I20260927 18:33:37.575214 11172 ts_tablet_manager.cc:1428] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 18:33:37.575618 11173 tablet_bootstrap.cc:492] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: No bootstrap required, opened a new log
I20260927 18:33:37.575555 11171 consensus_queue.cc:260] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.575939 11173 ts_tablet_manager.cc:1397] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260927 18:33:37.578383 11020 heartbeater.cc:503] Master 127.0.97.126:37519 was elected leader, sending a full tablet report...
I20260927 18:33:37.579366 11171 ts_tablet_manager.cc:1428] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 18:33:37.584450 11173 raft_consensus.cc:348] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.584769 11173 raft_consensus.cc:374] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:33:37.584873 11173 raft_consensus.cc:729] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93627729a0314b18a8e7fada44827f81, State: Initialized, Role: FOLLOWER
I20260927 18:33:37.585457 11173 consensus_queue.cc:260] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.587146 11173 ts_tablet_manager.cc:1428] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
W20260927 18:33:37.679549 10887 tablet.cc:2369] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:33:37.681018 11021 tablet.cc:2369] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:33:37.684010 11154 tablet.cc:2369] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:33:37.829347 11177 raft_consensus.cc:482] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:37.829699 11177 raft_consensus.cc:504] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.831502 11177 leader_election.cc:288] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbcb280c6a89438fa9e4fd9f5cdb949c (127.0.97.66:46637), 93627729a0314b18a8e7fada44827f81 (127.0.97.65:46433)
I20260927 18:33:37.838250 11179 raft_consensus.cc:482] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:33:37.838757 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.832059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44823 (local address 127.0.97.66:46637)
0927 18:33:37.832255 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.832260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.832282 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:33:37.832506 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.832513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.832589 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.832720 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.833355 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.834370 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.836571 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.837734 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.837979 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.838071 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.838335 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.838479 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:33:37.838796 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.831956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:46637 (local address 127.0.97.67:44823)
0927 18:33:37.832178 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.832197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.832330 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.832934 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.832949 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.832975 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.833151 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.833161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.834599 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.834605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.836410 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.836424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.837950 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.837959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.838152 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.838188 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.838294 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.838428 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:33:37.838634 11179 raft_consensus.cc:504] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.839998 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.832598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33945 (local address 127.0.97.65:46433)
0927 18:33:37.832836 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.832844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.833229 (+ 385us) server_negotiation.cc:407] Connection header received
0927 18:33:37.833368 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.833372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.833433 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.833510 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.834157 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.835382 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.837584 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.838806 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.839175 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.839286 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.839635 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.839775 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:33:37.840104 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.832413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46433 (local address 127.0.97.67:33945)
0927 18:33:37.833109 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.833134 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.833231 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.833556 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.833561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.833589 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.833953 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.833965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.835548 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.835552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.837351 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.837366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.839267 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.839274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.839460 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.839498 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.839597 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.839718 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":1,"thread_start_us":428,"threads_started":1}
I20260927 18:33:37.839918 10975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "7bde1788dbef43f98c449015d5561561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" is_pre_election: true
I20260927 18:33:37.840514 10975 raft_consensus.cc:2454] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bde1788dbef43f98c449015d5561561 in term 0.
I20260927 18:33:37.840754 11179 leader_election.cc:288] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbcb280c6a89438fa9e4fd9f5cdb949c (127.0.97.66:46637), 7bde1788dbef43f98c449015d5561561 (127.0.97.67:35097)
I20260927 18:33:37.841231 11042 leader_election.cc:302] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bde1788dbef43f98c449015d5561561, cbcb280c6a89438fa9e4fd9f5cdb949c; no voters:
I20260927 18:33:37.841696 11177 raft_consensus.cc:2781] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:33:37.841861 11177 raft_consensus.cc:482] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:33:37.840996 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "7bde1788dbef43f98c449015d5561561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93627729a0314b18a8e7fada44827f81" is_pre_election: true
I20260927 18:33:37.841993 11177 raft_consensus.cc:3037] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:37.842173 10841 raft_consensus.cc:2454] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bde1788dbef43f98c449015d5561561 in term 0.
I20260927 18:33:37.846901 11177 raft_consensus.cc:504] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.848235 10975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "7bde1788dbef43f98c449015d5561561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c"
I20260927 18:33:37.848486 10975 raft_consensus.cc:3037] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:37.848785 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "7bde1788dbef43f98c449015d5561561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93627729a0314b18a8e7fada44827f81"
I20260927 18:33:37.849026 10841 raft_consensus.cc:3037] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:33:37.852581 10975 raft_consensus.cc:2454] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bde1788dbef43f98c449015d5561561 in term 1.
I20260927 18:33:37.853040 11042 leader_election.cc:302] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bde1788dbef43f98c449015d5561561, cbcb280c6a89438fa9e4fd9f5cdb949c; no voters:
I20260927 18:33:37.853746 11183 raft_consensus.cc:2781] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:33:37.853924 11177 leader_election.cc:288] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [CANDIDATE]: Term 1 election: Requested vote from peers cbcb280c6a89438fa9e4fd9f5cdb949c (127.0.97.66:46637), 93627729a0314b18a8e7fada44827f81 (127.0.97.65:46433)
I20260927 18:33:37.854158 10841 raft_consensus.cc:2454] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bde1788dbef43f98c449015d5561561 in term 1.
I20260927 18:33:37.854707 11183 raft_consensus.cc:686] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 1 LEADER]: Becoming Leader. State: Replica: 7bde1788dbef43f98c449015d5561561, State: Running, Role: LEADER
I20260927 18:33:37.855325 11183 consensus_queue.cc:237] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } }
I20260927 18:33:37.860764 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.841399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33711 (local address 127.0.97.66:46637)
0927 18:33:37.841550 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.841557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.853402 (+ 11845us) server_negotiation.cc:407] Connection header received
0927 18:33:37.853469 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.853476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.853558 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.853641 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.854985 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.856046 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.858631 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.859788 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.859932 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.860051 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.860392 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.860513 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:33:37.860862 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.841052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:46637 (local address 127.0.97.65:33711)
0927 18:33:37.853129 (+ 12077us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.853145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.853314 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.854047 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.854052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.854078 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.854834 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.854847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.856195 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.856201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.858462 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.858477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.860050 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.860056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.860208 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.860239 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.860326 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.860455 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11938,"mutex_wait_us":294,"thread_start_us":11767,"threads_started":1}
I20260927 18:33:37.860718 10696 catalog_manager.cc:5495] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bde1788dbef43f98c449015d5561561 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "7bde1788dbef43f98c449015d5561561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bde1788dbef43f98c449015d5561561" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35097 } health_report { overall_health: HEALTHY } } }
I20260927 18:33:37.862186 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.841277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52613 (local address 127.0.97.67:35097)
0927 18:33:37.841478 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.841483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.853557 (+ 12074us) server_negotiation.cc:407] Connection header received
0927 18:33:37.853823 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.853828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.853923 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.853988 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:33:37.854773 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.856005 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.860188 (+ 4183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.861322 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.861469 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.861564 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.861837 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.861952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:33:37.862267 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.841458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:35097 (local address 127.0.97.65:52613)
0927 18:33:37.853463 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0927 18:33:37.853479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:33:37.853652 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:33:37.854167 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:33:37.854171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:33:37.854211 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:33:37.854567 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.854578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.856138 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.856144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.859987 (+ 3843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:33:37.860000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.861518 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.861524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:33:37.861667 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.861687 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:33:37.861783 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:33:37.861912 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11883,"mutex_wait_us":46,"thread_start_us":11969,"threads_started":1}
I20260927 18:33:37.863008 11108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "93627729a0314b18a8e7fada44827f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bde1788dbef43f98c449015d5561561" is_pre_election: true
I20260927 18:33:37.863049 10975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da96fdbc9d54a418d4cf5ee129597c8" candidate_uuid: "93627729a0314b18a8e7fada44827f81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" is_pre_election: true
I20260927 18:33:37.863371 10975 raft_consensus.cc:2379] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93627729a0314b18a8e7fada44827f81 in current term 1: Already voted for candidate 7bde1788dbef43f98c449015d5561561 in this term.
I20260927 18:33:37.864217 10775 leader_election.cc:302] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93627729a0314b18a8e7fada44827f81; no voters: 7bde1788dbef43f98c449015d5561561, cbcb280c6a89438fa9e4fd9f5cdb949c
I20260927 18:33:37.864583 11179 raft_consensus.cc:2726] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 18:33:37.898440 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.893471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.97.67:35097)
0927 18:33:37.893701 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.893705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.893800 (+ 95us) server_negotiation.cc:407] Connection header received
0927 18:33:37.894033 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.894036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.894124 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.894249 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:33:37.894535 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.895274 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.896532 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.896910 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.897007 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.897424 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:33:37.897960 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:33:37.898064 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.898162 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.898222 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 18:33:37.906034 10975 raft_consensus.cc:1261] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 1 FOLLOWER]: Refusing update from remote peer 7bde1788dbef43f98c449015d5561561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:37.906080 10841 raft_consensus.cc:1261] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 1 FOLLOWER]: Refusing update from remote peer 7bde1788dbef43f98c449015d5561561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:33:37.906852 11183 consensus_queue.cc:1035] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbcb280c6a89438fa9e4fd9f5cdb949c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:37.907189 11177 consensus_queue.cc:1035] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93627729a0314b18a8e7fada44827f81" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:33:37.918797 11186 mvcc.cc:205] Tried to move back new op lower bound from 7334027337333686272 to 7334027337140199424. Current Snapshot: MvccSnapshot[applied={T|T < 7334027337333686272}]
I20260927 18:33:37.933125 10697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:37.933418 10697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:33:37.938558 10697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5984480ece04c129d2a473960d1727f: Sending DeleteTablet for 3 replicas of tablet 3da96fdbc9d54a418d4cf5ee129597c8
I20260927 18:33:37.940167 10955 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34534
I20260927 18:33:37.940822 11088 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39130
I20260927 18:33:37.941133 10821 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da96fdbc9d54a418d4cf5ee129597c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37942
I20260927 18:33:37.946249 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10758
I20260927 18:33:37.951275 11197 tablet_replica.cc:330] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: stopping tablet replica
I20260927 18:33:37.951913 11197 raft_consensus.cc:2229] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:33:37.952566 11198 tablet_replica.cc:330] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: stopping tablet replica
I20260927 18:33:37.953253 11197 raft_consensus.cc:2258] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:37.955076 11198 raft_consensus.cc:2229] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:37.955588 11198 raft_consensus.cc:2258] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:37.958185 11197 ts_tablet_manager.cc:1905] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:37.958178 11198 ts_tablet_manager.cc:1905] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:37.965571 11197 ts_tablet_manager.cc:1918] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:37.965827 11197 log.cc:1196] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/3da96fdbc9d54a418d4cf5ee129597c8
I20260927 18:33:37.966446 11197 ts_tablet_manager.cc:1939] T 3da96fdbc9d54a418d4cf5ee129597c8 P 7bde1788dbef43f98c449015d5561561: Deleting consensus metadata
I20260927 18:33:37.968128 10684 catalog_manager.cc:4841] TS 7bde1788dbef43f98c449015d5561561 (127.0.97.67:35097): tablet 3da96fdbc9d54a418d4cf5ee129597c8 (table test-table [id=73ff53e53f8047e59a50b0293d774244]) successfully deleted
I20260927 18:33:37.969455 11196 tablet_replica.cc:330] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: stopping tablet replica
I20260927 18:33:37.969971 11196 raft_consensus.cc:2229] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:33:37.970566 11196 raft_consensus.cc:2258] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:33:37.972709 11196 ts_tablet_manager.cc:1905] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:33:37.973749 11198 ts_tablet_manager.cc:1918] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:37.973949 11198 log.cc:1196] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/3da96fdbc9d54a418d4cf5ee129597c8
I20260927 18:33:37.974449 11198 ts_tablet_manager.cc:1939] T 3da96fdbc9d54a418d4cf5ee129597c8 P 93627729a0314b18a8e7fada44827f81: Deleting consensus metadata
I20260927 18:33:37.976131 10682 catalog_manager.cc:4841] TS 93627729a0314b18a8e7fada44827f81 (127.0.97.65:46433): tablet 3da96fdbc9d54a418d4cf5ee129597c8 (table test-table [id=73ff53e53f8047e59a50b0293d774244]) successfully deleted
Sep 27 18:33:37 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:33:37.979456 11196 ts_tablet_manager.cc:1918] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:33:37.979602 11196 log.cc:1196] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/3da96fdbc9d54a418d4cf5ee129597c8
I20260927 18:33:37.980154 11196 ts_tablet_manager.cc:1939] T 3da96fdbc9d54a418d4cf5ee129597c8 P cbcb280c6a89438fa9e4fd9f5cdb949c: Deleting consensus metadata
I20260927 18:33:37.982008 10682 catalog_manager.cc:4841] TS cbcb280c6a89438fa9e4fd9f5cdb949c (127.0.97.66:46637): tablet 3da96fdbc9d54a418d4cf5ee129597c8 (table test-table [id=73ff53e53f8047e59a50b0293d774244]) successfully deleted
I20260927 18:33:37.984339 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:37.947562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.0.97.65:46433)
0927 18:33:37.947727 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:37.947733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:37.962957 (+ 15224us) server_negotiation.cc:407] Connection header received
0927 18:33:37.963133 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:37.963139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:37.963242 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:37.963347 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:37.965113 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.966299 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.968265 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:37.972594 (+ 4329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:37.972696 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:37.977895 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:37.977909 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:37.977928 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:37.977969 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:37.981690 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.982504 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.982510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.982512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.982627 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:37.983024 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:37.983026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:37.983028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:37.983351 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:37.983547 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:37.983898 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 18:33:37.984041 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:33:37.985000 10881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:38.092887 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10758
I20260927 18:33:38.111150 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10892
Sep 27 18:33:38 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:33:38.128005 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:38.111980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.97.66:46637)
0927 18:33:38.112177 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:38.112184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:38.112206 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:33:38.112460 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:38.112464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:38.112532 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:38.112651 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:38.113992 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:38.114693 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:38.116019 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:38.116462 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:38.116611 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:38.121138 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:38.121155 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:38.121175 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:38.121224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:38.124864 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:38.125511 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:38.125521 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:38.125525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:38.125683 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:38.126398 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:38.126404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:38.126408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:38.126824 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:38.127120 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:38.127451 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:33:38.127666 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:33:38.128778 11015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:38.218161 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10892
I20260927 18:33:38.236901 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 11025
Sep 27 18:33:38 dist-test-slave-n3p4 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:33:38.258857 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:33:38.237761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.0.97.67:35097)
0927 18:33:38.237997 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:33:38.238009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:33:38.238031 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:33:38.238228 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:33:38.238235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:33:38.238331 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:33:38.238467 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:33:38.239865 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:38.241008 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:38.242545 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:33:38.243260 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:33:38.243429 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:33:38.251749 (+ 8320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:33:38.251767 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:33:38.251788 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:33:38.251839 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:33:38.255935 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:38.256666 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:38.256676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:38.256682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:38.256838 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:33:38.257183 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:33:38.257190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:33:38.257195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:33:38.257729 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:33:38.258075 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 18:33:38.258241 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:33:38.258503 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:33:38.259631 11148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:38.382921 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 11025
I20260927 18:33:38.404044 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 10665
I20260927 18:33:38.405205 10727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:33:38.530762 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 10665
2026-09-27T18:33:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4505 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:33:38 dist-test-slave-n3p4 krb5kdc[11206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:33:38 dist-test-slave-n3p4 krb5kdc[11206](info): set up 2 sockets
Sep 27 18:33:38 dist-test-slave-n3p4 krb5kdc[11206](info): commencing operation
krb5kdc: starting...
W20260927 18:33:39.626794 389 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:33:39 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:33:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260927 18:33:39.937536 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKYOLso/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/postgres -l logfile start
2026-09-27 18:33:42.428 UTC [11236] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:33:42.428 UTC [11236] LOG: listening on IPv4 address "127.0.97.84", port 35417
2026-09-27 18:33:42.431 UTC [11236] LOG: listening on Unix socket "/tmp/.s.PGSQL.35417"
2026-09-27 18:33:42.453 UTC [11239] LOG: database system was shut down at 2026-09-27 18:33:41 UTC
2026-09-27 18:33:42.457 UTC [11236] LOG: database system is ready to accept connections
I20260927 18:33:43.434507 389 mini_postgres.cc:96] Postgres bound to 35417
2026-09-27 18:33:43.452 UTC [11247] FATAL: database "slave" does not exist
127.0.97.84:35417 - accepting connections
I20260927 18:33:43.452771 389 mini_ranger.cc:161] Starting Ranger...
I20260927 18:33:43.478475 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:33:44.247269 389 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:33:44.247406 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin
2026-09-27 18:33:44,527 [I] DB FLAVOR :POSTGRES
2026-09-27 18:33:44,528 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:33:44,528 [I] Checking connection..
2026-09-27 18:33:44,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:33:44,760 [I] Checking connection passed.
2026-09-27 18:33:44,761 [I] --------- Verifying version history table ---------
2026-09-27 18:33:44,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:33:45,002 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:33:45,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:33:45,257 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:33:45,258 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:33:45,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:33:45.490 UTC [11311] WARNING: there is no transaction in progress
2026-09-27 18:33:45,497 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:33:45,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:33:45,728 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:33:45,728 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:33:45,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:33:45,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:33:46,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3p4.c.gcp-upstream.internal','N') ;"
2026-09-27 18:33:46,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:33:46,679 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:33:46,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:33:46,939 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:33:46,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:33:47,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:33:47,452 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:33:47,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:33:48.402 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.419 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.436 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.452 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.468 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.486 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.555 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.563 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.578 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.588 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.598 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.612 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.622 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.628 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.635 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:48.641 UTC [11419] WARNING: there is no transaction in progress
2026-09-27 18:33:49,256 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:33:49,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:33:49,514 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:33:49,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:33:49,761 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:33:49,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:33:50,001 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:33:50,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:33:50,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:33:50,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3p4.c.gcp-upstream.internal';"
2026-09-27 18:33:50,710 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:33:50,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:33:50,949 [I] DB_PATCHES have already been applied
I20260927 18:33:50.955132 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKYOLso/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:33:50.955240 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260927 18:33:50.969600 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:44103
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:33:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:33:52.111012 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.658s user 0.004s sys 0.032s
Sep 27, 2026 6:34:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:34:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:34:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:34:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:34:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:34:12.689599 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:34:12 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534052, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:34:12.855264 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:34:12.875835 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:34:13.761474 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:34:13.761579 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms
I20260927 18:34:13.761695 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:35417
2026-09-27 18:34:14,161 [I] DB FLAVOR :POSTGRES
2026-09-27 18:34:14,161 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:34:14,162 [I] Checking connection
2026-09-27 18:34:14,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:34:14,409 [I] connection success
2026-09-27 18:34:14,409 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:34:14,409 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:34:14,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:34:14,646 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:34:14,646 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:34:14,647 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:34:14,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:35417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:34:14,895 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:34:15.361563 389 mini_ranger_kms.cc:310] Created kms service
I20260927 18:34:15.575479 389 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:34:15.626435 389 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:34:15 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534052, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:34:16.039798 389 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:34:16.040158 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:34:16.040230 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260927 18:34:16.058810 389 mini_ranger_kms.cc:276] Ranger KMS PID: 11599
I20260927 18:34:16.059015 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:42421
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:34:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:34:18.390715 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.535s user 0.087s sys 0.328s
I20260927 18:34:18.390971 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:34:18.391176 389 mini_ranger_kms.cc:401] 127.0.97.84:42421/kms/v1/keys
Sep 27 18:34:18 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:34:18 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:34:19 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:34:21 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:34:21.457449 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45603
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45603
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:42421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:21.825326 11650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:21.825606 11650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:21.825749 11650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:21.833024 11650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:21.833127 11650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:21.833216 11650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:21.833287 11650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:21.833380 11650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:21.843369 11650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:42421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45603
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:34:21.844635 11650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:21.846369 11650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:21.858515 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:21.858482 11656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:21.858933 11658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:21.859464 11650 server_base.cc:1034] running on GCE node
I20260927 18:34:21.860229 11650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:21.861588 11650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:21.863200 11650 hybrid_clock.cc:648] HybridClock initialized: now 1790534061862771 us; error 63 us; skew 500 ppm
I20260927 18:34:21.863540 11650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:34:21 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:21.866708 11650 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:34:21.868175 11650 webserver.cc:460] Webserver started at http://127.0.97.126:46729/ using document root <none> and password file <none>
I20260927 18:34:21.868825 11650 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:21.868937 11650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:21.869284 11650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:34:21 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:34:21.918146 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "d277a496b4bd4c15bfb557b4a95ef5f7"
format_stamp: "Formatted at 2026-09-27 18:34:21 on dist-test-slave-n3p4"
server_key: "f240a7aa9a33a66d24a360b132118456"
server_key_iv: "5e4a35f633a94d9892bd4df2ffaa11df"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:21.919179 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "d277a496b4bd4c15bfb557b4a95ef5f7"
format_stamp: "Formatted at 2026-09-27 18:34:21 on dist-test-slave-n3p4"
server_key: "f240a7aa9a33a66d24a360b132118456"
server_key_iv: "5e4a35f633a94d9892bd4df2ffaa11df"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:21.926383 11650 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 18:34:21.946133 11665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:21.947965 11650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:34:21.948138 11650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "d277a496b4bd4c15bfb557b4a95ef5f7"
format_stamp: "Formatted at 2026-09-27 18:34:21 on dist-test-slave-n3p4"
server_key: "f240a7aa9a33a66d24a360b132118456"
server_key_iv: "5e4a35f633a94d9892bd4df2ffaa11df"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:21.948376 11650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:21.995810 11650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:22.000267 11650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:22.000552 11650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:22.020071 11650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45603
I20260927 18:34:22.020071 11716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45603 every 8 connection(s)
I20260927 18:34:22.021425 11650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:22.025549 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:22.025624 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 11650
I20260927 18:34:22.025843 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:34:22.026695 389 external_mini_cluster.cc:1428] Setting key d86a8d80b0198c470e894a9b183bae7c
I20260927 18:34:22.044147 11717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: Bootstrap starting.
I20260927 18:34:22.047902 11717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:22.049175 11717 log.cc:824] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:22.052577 11717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: No bootstrap required, opened a new log
Sep 27 18:34:22 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:34:22.062012 11717 raft_consensus.cc:348] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } }
I20260927 18:34:22.062490 11717 raft_consensus.cc:374] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:22.062598 11717 raft_consensus.cc:729] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d277a496b4bd4c15bfb557b4a95ef5f7, State: Initialized, Role: FOLLOWER
I20260927 18:34:22.063308 11717 consensus_queue.cc:260] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } }
I20260927 18:34:22.063556 11717 raft_consensus.cc:388] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:22.063720 11717 raft_consensus.cc:482] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:22.063901 11717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:22.066089 11717 raft_consensus.cc:504] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } }
I20260927 18:34:22.066644 11717 leader_election.cc:302] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d277a496b4bd4c15bfb557b4a95ef5f7; no voters:
I20260927 18:34:22.067297 11717 leader_election.cc:288] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:22.067397 11722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:22.067870 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:22.028673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.0.97.126:45603)
0927 18:34:22.042392 (+ 13719us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:22.042409 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:22.042943 (+ 534us) server_negotiation.cc:407] Connection header received
0927 18:34:22.045390 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:22.045416 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:22.045999 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:22.046462 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:22.047881 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:22.049951 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:22.051581 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:22.052690 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:22.052909 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:22.057801 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:22.057812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:22.057835 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:22.057880 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:22.061901 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:22.062734 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:22.062742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:22.062751 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:22.062919 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:22.063356 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:22.063361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:22.063365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:22.064057 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:22.065393 (+ 1336us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:22.066134 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 18:34:22.066550 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13403,"thread_start_us":13169,"threads_started":1}
I20260927 18:34:22.074501 11722 raft_consensus.cc:686] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [term 1 LEADER]: Becoming Leader. State: Replica: d277a496b4bd4c15bfb557b4a95ef5f7, State: Running, Role: LEADER
I20260927 18:34:22.075065 11722 consensus_queue.cc:237] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } }
I20260927 18:34:22.075990 11717 sys_catalog.cc:564] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:22.082095 11723 sys_catalog.cc:455] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } } }
I20260927 18:34:22.082516 11723 sys_catalog.cc:458] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:22.083427 11724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d277a496b4bd4c15bfb557b4a95ef5f7. Latest consensus state: current_term: 1 leader_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d277a496b4bd4c15bfb557b4a95ef5f7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45603 } } }
I20260927 18:34:22.083766 11724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:22.093916 11731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:22.100965 11731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:22.122890 11717 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:34:22.133924 11731 catalog_manager.cc:1347] Generated new cluster ID: cfd2032453ac4c96a957357cc5a47774
I20260927 18:34:22.134173 11731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:22.146992 11731 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:22.149039 11731 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:22.169691 11731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: Generated new TSK 0
I20260927 18:34:22.172147 11731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:34:22.967370 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:34:23 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:34:23.392952 11762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:23.393337 11762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:23.393540 11762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:23.405242 11762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:23.405437 11762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:23.405649 11762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:34:23.420554 11762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:34:23.422271 11762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:23.424027 11762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:23.444335 11772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.445622 11773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.452023 11762 server_base.cc:1034] running on GCE node
W20260927 18:34:23.452242 11776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.453171 11762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:23.454591 11762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:23.455891 11762 hybrid_clock.cc:648] HybridClock initialized: now 1790534063455821 us; error 56 us; skew 500 ppm
I20260927 18:34:23.456420 11762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:34:23 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:23.461063 11762 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:34:23.463105 11762 webserver.cc:460] Webserver started at http://127.0.97.65:44345/ using document root <none> and password file <none>
I20260927 18:34:23.464038 11762 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:23.464175 11762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:23.464533 11762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:34:23 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:34:23.487052 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "226bcadc6aa641f99d8ba1d586a56068"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-n3p4"
server_key: "5a91d98bf46f69c1f99145c8d3199d3b"
server_key_iv: "198ac80585dceeae404efcc32b4da043"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:23.488706 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "226bcadc6aa641f99d8ba1d586a56068"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-n3p4"
server_key: "5a91d98bf46f69c1f99145c8d3199d3b"
server_key_iv: "198ac80585dceeae404efcc32b4da043"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:23.516645 11762 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.014s sys 0.011s
I20260927 18:34:23.553442 11782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.559516 11762 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.010s sys 0.005s
I20260927 18:34:23.559751 11762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "226bcadc6aa641f99d8ba1d586a56068"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-n3p4"
server_key: "5a91d98bf46f69c1f99145c8d3199d3b"
server_key_iv: "198ac80585dceeae404efcc32b4da043"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:23.560000 11762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:23.583171 11762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:23.588137 11762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:23.588984 11762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:23.606263 11762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:23.609766 11762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:23.609977 11762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:23.610220 11762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:23.610363 11762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.698788 11762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41317
I20260927 18:34:23.698913 11894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41317 every 8 connection(s)
I20260927 18:34:23.700713 11762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:23.709513 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 11762
I20260927 18:34:23.709753 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:23.710738 389 external_mini_cluster.cc:1428] Setting key 70bbf3a1de4543ebd3bb6fe2f933b711
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Sep 27 18:34:23 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:34:23 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:34:23.779327 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.703876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43307 (local address 127.0.97.126:45603)
0927 18:34:23.724102 (+ 20226us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.724111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.725336 (+ 1225us) server_negotiation.cc:407] Connection header received
0927 18:34:23.728911 (+ 3575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.728917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.729011 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.729143 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.748353 (+ 19210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.749571 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.754627 (+ 5056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.755287 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.755428 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.770822 (+ 15394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.770831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.770836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.770899 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.774902 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.775899 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.775905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.775909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.776098 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.776633 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.776639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.776643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.777062 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.777233 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.778144 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.778390 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20044,"thread_start_us":21459,"threads_started":1}
I20260927 18:34:23.779896 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.704319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45603 (local address 127.0.97.65:43307)
0927 18:34:23.724837 (+ 20518us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.724901 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.728546 (+ 3645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.729484 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.729502 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.745471 (+ 15969us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.748065 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.748106 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.751951 (+ 3845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.751958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.754435 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.754455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.755573 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.755582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.755902 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.766262 (+ 10360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.766307 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.770499 (+ 4192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.775103 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.775114 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.775135 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.775736 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.776227 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.776232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.776235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.776494 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.777199 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.777208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.777652 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.777999 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.778535 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20114,"thread_start_us":19845,"threads_started":1}
I20260927 18:34:23.782015 11895 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45603
I20260927 18:34:23.782352 11895 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:23.783140 11895 heartbeater.cc:511] Master 127.0.97.126:45603 requested a full tablet report, sending...
I20260927 18:34:23.785910 11682 ts_manager.cc:194] Registered new tserver with Master: 226bcadc6aa641f99d8ba1d586a56068 (127.0.97.65:41317)
I20260927 18:34:23.788019 11682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:43307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:34:23.814435 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260927 18:34:24.239238 11901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:24.239704 11901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:24.239881 11901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:24.250149 11901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:24.250291 11901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:24.250469 11901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:34:24.262758 11901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:34:24.264156 11901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:24.265793 11901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:24.301007 11909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:24.300989 11908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:24.304000 11901 server_base.cc:1034] running on GCE node
W20260927 18:34:24.305691 11911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:24.306676 11901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:24.315006 11901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:24.316304 11901 hybrid_clock.cc:648] HybridClock initialized: now 1790534064316194 us; error 111 us; skew 500 ppm
I20260927 18:34:24.316771 11901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:24.321419 11901 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:34:24.330050 11901 webserver.cc:460] Webserver started at http://127.0.97.66:36727/ using document root <none> and password file <none>
I20260927 18:34:24.330827 11901 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:24.330978 11901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:24.331408 11901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:34:24.350667 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "2ee72bf0801849a3a1664c46bd9e2d30"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "66766272f83b3ab7b462c86874646cbd"
server_key_iv: "b1e9ca0bd2474c7be7b58daacbc6a393"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.351850 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "2ee72bf0801849a3a1664c46bd9e2d30"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "66766272f83b3ab7b462c86874646cbd"
server_key_iv: "b1e9ca0bd2474c7be7b58daacbc6a393"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.359504 11901 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 18:34:24.398363 11925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:24.400288 11901 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:34:24.400465 11901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "2ee72bf0801849a3a1664c46bd9e2d30"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "66766272f83b3ab7b462c86874646cbd"
server_key_iv: "b1e9ca0bd2474c7be7b58daacbc6a393"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.400720 11901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:24.442281 11901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:24.446239 11901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:24.446518 11901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:24.447491 11901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:24.448814 11901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:24.448889 11901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:24.449020 11901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:24.449070 11901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:24.482986 11901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37831
I20260927 18:34:24.483017 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37831 every 8 connection(s)
I20260927 18:34:24.484288 11901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:24.488240 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 11901
I20260927 18:34:24.488412 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:24.488967 389 external_mini_cluster.cc:1428] Setting key 4c5c4858d211109d9e48e2425e4e4697
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:34:24.506121 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:24.487539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58887 (local address 127.0.97.126:45603)
0927 18:34:24.488085 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:24.488094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:24.489267 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 18:34:24.491283 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:24.491289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:24.491384 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:24.491516 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:24.492872 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.494096 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.495628 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.496343 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.496478 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:24.499916 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:24.499920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:24.499924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:24.499955 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:24.502671 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:24.503542 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:24.503546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:24.503548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:24.503656 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:24.504157 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:24.504161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:24.504164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:24.504474 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:24.504603 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:24.505524 (+ 921us) server_negotiation.cc:299] Negotiation successful
0927 18:34:24.505714 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":218,"threads_started":1}
I20260927 18:34:24.507445 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:24.488128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45603 (local address 127.0.97.66:58887)
0927 18:34:24.489039 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:24.489084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:24.491000 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:24.491702 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:24.491711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:24.492120 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:24.492709 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:24.492726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.494230 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.494234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:24.495493 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:24.495505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.496540 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.496544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:24.496721 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:24.497426 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:24.497446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:24.499721 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:24.502821 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:24.502833 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:24.502853 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:24.503440 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:24.503764 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:24.503772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:24.503779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:24.504020 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:24.504619 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:24.504631 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:24.505036 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 18:34:24.505372 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 18:34:24.505964 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":434,"threads_started":1}
I20260927 18:34:24.509482 12038 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45603
I20260927 18:34:24.509850 12038 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:24.510644 12038 heartbeater.cc:511] Master 127.0.97.126:45603 requested a full tablet report, sending...
I20260927 18:34:24.512234 11682 ts_manager.cc:194] Registered new tserver with Master: 2ee72bf0801849a3a1664c46bd9e2d30 (127.0.97.66:37831)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:34:24.513127 11682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:58887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:34:24.541342 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:34:24.791877 11895 heartbeater.cc:503] Master 127.0.97.126:45603 was elected leader, sending a full tablet report...
W20260927 18:34:24.875272 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:24.875537 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:24.875694 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:24.883157 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:24.883263 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:24.883419 12044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:34:24.893509 12044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:34:24.894747 12044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:24.895982 12044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:24.909436 12049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:24.909724 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:24.909718 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:24.910940 12044 server_base.cc:1034] running on GCE node
I20260927 18:34:24.911505 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:24.912461 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:24.913667 12044 hybrid_clock.cc:648] HybridClock initialized: now 1790534064913628 us; error 50 us; skew 500 ppm
I20260927 18:34:24.914008 12044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:24.916963 12044 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:34:24.918460 12044 webserver.cc:460] Webserver started at http://127.0.97.67:42871/ using document root <none> and password file <none>
I20260927 18:34:24.919099 12044 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:24.919241 12044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:24.919548 12044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:34:24 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:34:24.934715 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "421116b81eab42e280a341586c32b1fe"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "7ff915b5e7d133bbd0fb60bc6ca8f1b3"
server_key_iv: "0d13d24a89e83ba2b890f55442611812"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.935539 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "421116b81eab42e280a341586c32b1fe"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "7ff915b5e7d133bbd0fb60bc6ca8f1b3"
server_key_iv: "0d13d24a89e83ba2b890f55442611812"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.941170 12044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:34:24.957103 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:24.958643 12044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:34:24.958819 12044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "421116b81eab42e280a341586c32b1fe"
format_stamp: "Formatted at 2026-09-27 18:34:24 on dist-test-slave-n3p4"
server_key: "7ff915b5e7d133bbd0fb60bc6ca8f1b3"
server_key_iv: "0d13d24a89e83ba2b890f55442611812"
server_key_version: "kuduclusterkey@0"
I20260927 18:34:24.959031 12044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:24.975903 12044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:24.979660 12044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:24.979941 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:24.980800 12044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:24.982442 12044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:24.982537 12044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:24.982674 12044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:24.982740 12044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:25.016969 12044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35311
I20260927 18:34:25.017004 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35311 every 8 connection(s)
I20260927 18:34:25.018362 12044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:25.024417 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 12044
I20260927 18:34:25.024576 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:25.025066 389 external_mini_cluster.cc:1428] Setting key 55d33f9fcdfb1991fad14a964682db99
Sep 27 18:34:25 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:34:25.040786 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.023094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60159 (local address 127.0.97.126:45603)
0927 18:34:25.023610 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.023616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.025078 (+ 1462us) server_negotiation.cc:407] Connection header received
0927 18:34:25.027526 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.027531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.027687 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.027821 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.029448 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.030251 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.031693 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.032113 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.032203 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.035387 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.035392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.035397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.035430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.037766 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.038609 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.038618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.038621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.038746 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.039152 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.039159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.039162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.039440 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.039581 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.040233 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.040440 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
I20260927 18:34:25.041687 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.023751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45603 (local address 127.0.97.67:60159)
0927 18:34:25.024517 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.024564 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.027240 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.028071 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.028083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.028620 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:25.029302 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.029320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.030380 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.030384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.031573 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.031584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.032290 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.032295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.032481 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.033151 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:25.033172 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:25.035191 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:25.037921 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:25.037927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:25.037939 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:25.038478 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:25.038852 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:25.038858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:25.038862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:25.039049 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:25.039572 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:25.039580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:25.039878 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.040134 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.040602 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":318,"threads_started":1}
I20260927 18:34:25.042899 12172 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45603
I20260927 18:34:25.043238 12172 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:25.043928 12172 heartbeater.cc:511] Master 127.0.97.126:45603 requested a full tablet report, sending...
I20260927 18:34:25.045264 11682 ts_manager.cc:194] Registered new tserver with Master: 421116b81eab42e280a341586c32b1fe (127.0.97.67:35311)
I20260927 18:34:25.046123 11682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:60159
I20260927 18:34:25.051988 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:34:25.069914 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.057520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.97.126:45603)
0927 18:34:25.057674 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.057678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.058139 (+ 461us) server_negotiation.cc:407] Connection header received
0927 18:34:25.058239 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.058243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.058328 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.058410 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.059865 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.060662 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.062128 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.062546 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.062670 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.064970 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.064974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.064978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.065007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.067733 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.068352 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.068355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.068357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.068478 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.068779 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.068782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.068785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.069024 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.069231 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.069467 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.069623 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:34:25.073846 11682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:25.081261 11682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:25.108356 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.093055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41317 (local address 127.0.0.1:36328)
0927 18:34:25.094646 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.094699 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.094905 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.097127 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.097136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.097174 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.097646 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.097665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.100329 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.100338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.103546 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.103566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.107004 (+ 3438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.107014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.107324 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.107402 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.107554 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.107818 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"threads_started":1}
I20260927 18:34:25.109647 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.093290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.0.97.65:41317)
0927 18:34:25.096120 (+ 2830us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.096130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.096163 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:34:25.096291 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.096301 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.096580 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.096792 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.097975 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.100113 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.103779 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.105813 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.105986 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.106307 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.108858 (+ 2551us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.109154 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2646,"thread_start_us":2047,"threads_started":1}
I20260927 18:34:25.115899 11830 tablet_service.cc:1467] Processing CreateTablet for tablet 6184d1d9ec56442a8375f2013d5c858c (DEFAULT_TABLE table=test-table [id=21f51a5859674c329e55adad6f67881d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.117615 11830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6184d1d9ec56442a8375f2013d5c858c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.118927 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.093449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37831 (local address 127.0.0.1:34550)
0927 18:34:25.110290 (+ 16841us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.110316 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.110471 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.112206 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.112212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.112243 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.112488 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.112499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.114505 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.114509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.116906 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.116917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.118155 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.118161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.118334 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.118393 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.118491 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.118612 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16657,"mutex_wait_us":21,"spinlock_wait_cycles":4992,"thread_start_us":923,"threads_started":1}
I20260927 18:34:25.119074 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.97.66:37831)
0927 18:34:25.111609 (+ 17451us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.111620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.111648 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:25.111744 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.111750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.111981 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.112463 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.112703 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.114349 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.117042 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.118010 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.118145 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.118322 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.118530 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.118739 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17269,"thread_start_us":17134,"threads_started":1}
I20260927 18:34:25.119975 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.096747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:35311 (local address 127.0.0.1:55358)
0927 18:34:25.111300 (+ 14553us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.111327 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.111529 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.112239 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.112243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.112278 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.112889 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.112900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.114182 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.114190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.116784 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.116799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.119080 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.119087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.119286 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.119326 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.119429 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.119555 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14399,"mutex_wait_us":102,"spinlock_wait_cycles":896,"thread_start_us":14264,"threads_started":1}
I20260927 18:34:25.120550 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.097316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.0.97.67:35311)
0927 18:34:25.106800 (+ 9484us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.106813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.111426 (+ 4613us) server_negotiation.cc:407] Connection header received
0927 18:34:25.111723 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.111732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.111991 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.112159 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.113044 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.114061 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.116939 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.118553 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.118705 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.118937 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.120059 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.120219 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9311,"thread_start_us":9150,"threads_started":1}
I20260927 18:34:25.122344 11973 tablet_service.cc:1467] Processing CreateTablet for tablet 6184d1d9ec56442a8375f2013d5c858c (DEFAULT_TABLE table=test-table [id=21f51a5859674c329e55adad6f67881d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.123867 11973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6184d1d9ec56442a8375f2013d5c858c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.124267 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 6184d1d9ec56442a8375f2013d5c858c (DEFAULT_TABLE table=test-table [id=21f51a5859674c329e55adad6f67881d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.125608 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6184d1d9ec56442a8375f2013d5c858c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.130654 12191 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Bootstrap starting.
I20260927 18:34:25.134050 12191 tablet_bootstrap.cc:654] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.135517 12191 log.cc:824] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:25.138126 12191 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: No bootstrap required, opened a new log
I20260927 18:34:25.138423 12191 ts_tablet_manager.cc:1397] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 18:34:25.139096 12193 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Bootstrap starting.
I20260927 18:34:25.139884 12194 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Bootstrap starting.
I20260927 18:34:25.143400 12193 tablet_bootstrap.cc:654] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.144485 12194 tablet_bootstrap.cc:654] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.145738 12194 log.cc:824] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:25.145823 12193 log.cc:824] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:25.149629 12194 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: No bootstrap required, opened a new log
I20260927 18:34:25.149657 12193 tablet_bootstrap.cc:492] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: No bootstrap required, opened a new log
I20260927 18:34:25.149932 12194 ts_tablet_manager.cc:1397] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260927 18:34:25.149950 12193 ts_tablet_manager.cc:1397] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260927 18:34:25.150867 12191 raft_consensus.cc:348] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.151343 12191 raft_consensus.cc:374] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.151469 12191 raft_consensus.cc:729] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226bcadc6aa641f99d8ba1d586a56068, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.152136 12191 consensus_queue.cc:260] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.154096 12191 ts_tablet_manager.cc:1428] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 18:34:25.160327 12194 raft_consensus.cc:348] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.160704 12194 raft_consensus.cc:374] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.160826 12194 raft_consensus.cc:729] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421116b81eab42e280a341586c32b1fe, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.160811 12193 raft_consensus.cc:348] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.161253 12193 raft_consensus.cc:374] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.161352 12193 raft_consensus.cc:729] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee72bf0801849a3a1664c46bd9e2d30, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.161480 12194 consensus_queue.cc:260] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.162045 12193 consensus_queue.cc:260] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.162711 12172 heartbeater.cc:503] Master 127.0.97.126:45603 was elected leader, sending a full tablet report...
I20260927 18:34:25.163134 12038 heartbeater.cc:503] Master 127.0.97.126:45603 was elected leader, sending a full tablet report...
I20260927 18:34:25.163529 12194 ts_tablet_manager.cc:1428] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 18:34:25.163846 12193 ts_tablet_manager.cc:1428] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260927 18:34:25.209512 11896 tablet.cc:2369] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:25.236904 12039 tablet.cc:2369] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:25.269942 12173 tablet.cc:2369] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:25.396448 12198 raft_consensus.cc:482] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:25.396725 12198 raft_consensus.cc:504] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.398409 12198 leader_election.cc:288] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 226bcadc6aa641f99d8ba1d586a56068 (127.0.97.65:41317), 2ee72bf0801849a3a1664c46bd9e2d30 (127.0.97.66:37831)
I20260927 18:34:25.405711 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.398840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50131 (local address 127.0.97.65:41317)
0927 18:34:25.399025 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.399031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.399084 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:34:25.399284 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.399289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.399353 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.399433 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.400497 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.401546 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.403644 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.404871 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.404967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.405039 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.405429 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.405535 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:34:25.405864 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.398826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41317 (local address 127.0.97.67:50131)
0927 18:34:25.398984 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.399005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.399119 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.399501 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.399510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.399558 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.400293 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.400310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.401709 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.401716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.403484 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.403497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.405018 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.405025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.405224 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.405264 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.405385 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.405523 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:34:25.406919 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.398828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53319 (local address 127.0.97.66:37831)
0927 18:34:25.399022 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.399029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.400562 (+ 1533us) server_negotiation.cc:407] Connection header received
0927 18:34:25.400778 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.400783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.400847 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.400935 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.401485 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.402648 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.404692 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.406018 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.406154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.406305 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.406542 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.406665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:34:25.406991 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.398847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37831 (local address 127.0.97.67:53319)
0927 18:34:25.400444 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.400462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.400587 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.400967 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.400972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.400996 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.401318 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.401329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.402816 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.402819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.404525 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.404539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.406266 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.406272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.406381 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.406403 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.406532 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.406648 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":359,"thread_start_us":873,"threads_started":1}
I20260927 18:34:25.406952 11850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6184d1d9ec56442a8375f2013d5c858c" candidate_uuid: "421116b81eab42e280a341586c32b1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226bcadc6aa641f99d8ba1d586a56068" is_pre_election: true
I20260927 18:34:25.407555 11850 raft_consensus.cc:2454] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421116b81eab42e280a341586c32b1fe in term 0.
I20260927 18:34:25.407840 11993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6184d1d9ec56442a8375f2013d5c858c" candidate_uuid: "421116b81eab42e280a341586c32b1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" is_pre_election: true
I20260927 18:34:25.408213 12063 leader_election.cc:302] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226bcadc6aa641f99d8ba1d586a56068, 421116b81eab42e280a341586c32b1fe; no voters:
I20260927 18:34:25.408303 11993 raft_consensus.cc:2454] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421116b81eab42e280a341586c32b1fe in term 0.
I20260927 18:34:25.408609 12198 raft_consensus.cc:2781] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:25.408771 12198 raft_consensus.cc:482] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:25.408906 12198 raft_consensus.cc:3037] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.411207 12198 raft_consensus.cc:504] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.411871 12198 leader_election.cc:288] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [CANDIDATE]: Term 1 election: Requested vote from peers 226bcadc6aa641f99d8ba1d586a56068 (127.0.97.65:41317), 2ee72bf0801849a3a1664c46bd9e2d30 (127.0.97.66:37831)
I20260927 18:34:25.412310 11850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6184d1d9ec56442a8375f2013d5c858c" candidate_uuid: "421116b81eab42e280a341586c32b1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226bcadc6aa641f99d8ba1d586a56068"
I20260927 18:34:25.412808 11993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6184d1d9ec56442a8375f2013d5c858c" candidate_uuid: "421116b81eab42e280a341586c32b1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee72bf0801849a3a1664c46bd9e2d30"
I20260927 18:34:25.412961 11850 raft_consensus.cc:3037] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.412992 11993 raft_consensus.cc:3037] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.416230 11993 raft_consensus.cc:2454] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421116b81eab42e280a341586c32b1fe in term 1.
I20260927 18:34:25.416232 11850 raft_consensus.cc:2454] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421116b81eab42e280a341586c32b1fe in term 1.
I20260927 18:34:25.416632 12063 leader_election.cc:302] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226bcadc6aa641f99d8ba1d586a56068, 421116b81eab42e280a341586c32b1fe; no voters:
I20260927 18:34:25.416942 12198 raft_consensus.cc:2781] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:25.417394 12198 raft_consensus.cc:686] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 1 LEADER]: Becoming Leader. State: Replica: 421116b81eab42e280a341586c32b1fe, State: Running, Role: LEADER
I20260927 18:34:25.417958 12198 consensus_queue.cc:237] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } }
I20260927 18:34:25.422351 11682 catalog_manager.cc:5495] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 421116b81eab42e280a341586c32b1fe (127.0.97.67). New cstate: current_term: 1 leader_uuid: "421116b81eab42e280a341586c32b1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "421116b81eab42e280a341586c32b1fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35311 } health_report { overall_health: HEALTHY } } }
I20260927 18:34:25.469085 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.462929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.97.67:35311)
0927 18:34:25.463119 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.463126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.463324 (+ 198us) server_negotiation.cc:407] Connection header received
0927 18:34:25.463541 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.463545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.463691 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.463839 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:25.464214 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.465010 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.466587 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.466983 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.467115 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.468008 (+ 893us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.468523 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.468695 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.468758 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.468832 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:34:25.477114 11993 raft_consensus.cc:1261] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 1 FOLLOWER]: Refusing update from remote peer 421116b81eab42e280a341586c32b1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.477279 11850 raft_consensus.cc:1261] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 1 FOLLOWER]: Refusing update from remote peer 421116b81eab42e280a341586c32b1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.478144 12201 consensus_queue.cc:1035] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee72bf0801849a3a1664c46bd9e2d30" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:25.478485 12198 consensus_queue.cc:1035] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "226bcadc6aa641f99d8ba1d586a56068" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:25.497490 12204 mvcc.cc:205] Tried to move back new op lower bound from 7334027532184645632 to 7334027531955634176. Current Snapshot: MvccSnapshot[applied={T|T < 7334027532184645632}]
I20260927 18:34:25.515403 11682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.515676 11682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.519301 11682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d277a496b4bd4c15bfb557b4a95ef5f7: Sending DeleteTablet for 3 replicas of tablet 6184d1d9ec56442a8375f2013d5c858c
I20260927 18:34:25.520377 11830 tablet_service.cc:1514] Processing DeleteTablet for tablet 6184d1d9ec56442a8375f2013d5c858c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36328
I20260927 18:34:25.520439 11973 tablet_service.cc:1514] Processing DeleteTablet for tablet 6184d1d9ec56442a8375f2013d5c858c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34550
I20260927 18:34:25.521796 12107 tablet_service.cc:1514] Processing DeleteTablet for tablet 6184d1d9ec56442a8375f2013d5c858c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55358
I20260927 18:34:25.521997 12213 tablet_replica.cc:330] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: stopping tablet replica
I20260927 18:34:25.522286 12214 tablet_replica.cc:330] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: stopping tablet replica
I20260927 18:34:25.522481 12213 raft_consensus.cc:2229] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.522761 12214 raft_consensus.cc:2229] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.523178 12213 raft_consensus.cc:2258] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.523283 12214 raft_consensus.cc:2258] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.523727 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 11762
I20260927 18:34:25.526082 12214 ts_tablet_manager.cc:1905] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.530532 12213 ts_tablet_manager.cc:1905] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.535569 12214 ts_tablet_manager.cc:1918] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:25.535842 12214 log.cc:1196] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/6184d1d9ec56442a8375f2013d5c858c
I20260927 18:34:25.536605 12214 ts_tablet_manager.cc:1939] T 6184d1d9ec56442a8375f2013d5c858c P 2ee72bf0801849a3a1664c46bd9e2d30: Deleting consensus metadata
Sep 27 18:34:25 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:34:25.537954 12213 ts_tablet_manager.cc:1918] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:25.538108 12213 log.cc:1196] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/6184d1d9ec56442a8375f2013d5c858c
I20260927 18:34:25.538658 12213 ts_tablet_manager.cc:1939] T 6184d1d9ec56442a8375f2013d5c858c P 226bcadc6aa641f99d8ba1d586a56068: Deleting consensus metadata
I20260927 18:34:25.539073 11668 catalog_manager.cc:4841] TS 2ee72bf0801849a3a1664c46bd9e2d30 (127.0.97.66:37831): tablet 6184d1d9ec56442a8375f2013d5c858c (table test-table [id=21f51a5859674c329e55adad6f67881d]) successfully deleted
I20260927 18:34:25.539453 12215 tablet_replica.cc:330] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: stopping tablet replica
I20260927 18:34:25.539973 12215 raft_consensus.cc:2229] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:25.540733 12215 raft_consensus.cc:2258] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.541941 11669 catalog_manager.cc:4841] TS 226bcadc6aa641f99d8ba1d586a56068 (127.0.97.65:41317): tablet 6184d1d9ec56442a8375f2013d5c858c (table test-table [id=21f51a5859674c329e55adad6f67881d]) successfully deleted
I20260927 18:34:25.543080 12215 ts_tablet_manager.cc:1905] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.547422 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.525407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.97.65:41317)
0927 18:34:25.525582 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.525588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.526881 (+ 1293us) server_negotiation.cc:407] Connection header received
0927 18:34:25.527104 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.527109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.527205 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.527309 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.529282 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.530394 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.532548 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.533286 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.533394 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.539556 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.539571 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.539590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.539631 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.543816 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.544660 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.544670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.544674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.544843 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.545293 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.545298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.545301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.545820 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.546131 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.546511 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.546755 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:34:25.548085 11890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:34:25.551342 12215 ts_tablet_manager.cc:1918] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:25.551560 12215 log.cc:1196] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/6184d1d9ec56442a8375f2013d5c858c
I20260927 18:34:25.552136 12215 ts_tablet_manager.cc:1939] T 6184d1d9ec56442a8375f2013d5c858c P 421116b81eab42e280a341586c32b1fe: Deleting consensus metadata
I20260927 18:34:25.553923 11668 catalog_manager.cc:4841] TS 421116b81eab42e280a341586c32b1fe (127.0.97.67:35311): tablet 6184d1d9ec56442a8375f2013d5c858c (table test-table [id=21f51a5859674c329e55adad6f67881d]) successfully deleted
I20260927 18:34:25.639426 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 11762
I20260927 18:34:25.659407 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 11901
Sep 27 18:34:25 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:34:25.678543 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.660334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.97.66:37831)
0927 18:34:25.660544 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.660551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.660573 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:34:25.660642 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.660648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.660880 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.661016 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.662660 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.663880 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.665225 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.665875 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.666019 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.670656 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.670698 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.670720 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.670770 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.674729 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.675454 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.675465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.675469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.675676 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.676127 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.676133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.676137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.676644 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.677001 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.677645 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.678008 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:34:25.679379 12033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:34:25.774083 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 11901
I20260927 18:34:25.791774 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 12044
Sep 27 18:34:25 dist-test-slave-n3p4 krb5kdc[11206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:34:25.810122 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.792811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.0.97.67:35311)
0927 18:34:25.793000 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.793007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.793030 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:34:25.793170 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.793175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.793368 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.793522 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.794963 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.796287 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.797556 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.798492 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.798648 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.802846 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.802861 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.802881 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.802947 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.807017 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.807742 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.807758 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.807764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.807978 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.808376 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.808382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.808386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.808865 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.809240 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.809432 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.809647 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:34:25.810865 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:34:25.931864 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 12044
I20260927 18:34:25.951207 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 11650
I20260927 18:34:25.952365 11712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:34:26.064137 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 11650
I20260927 18:34:26.081439 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:34:26.416554 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:34:26.416780 389 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:34:26.762002 389 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:34:26.762 UTC [11236] LOG: received smart shutdown request
2026-09-27 18:34:26.765 UTC [11236] LOG: background worker "logical replication launcher" (PID 11245) exited with exit code 1
2026-09-27 18:34:26.765 UTC [11240] LOG: shutting down
2026-09-27 18:34:26.922 UTC [11236] LOG: database system is shut down
2026-09-27T18:34:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48509 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:27 dist-test-slave-n3p4 krb5kdc[12229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:27 dist-test-slave-n3p4 krb5kdc[12229](info): set up 2 sockets
Sep 27 18:34:27 dist-test-slave-n3p4 krb5kdc[12229](info): commencing operation
krb5kdc: starting...
W20260927 18:34:28.146086 389 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:34:28 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260927 18:34:28.454684 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKYOLso/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/postgres -l logfile start
2026-09-27 18:34:31.069 UTC [12259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:34:31.069 UTC [12259] LOG: listening on IPv4 address "127.0.97.84", port 38243
2026-09-27 18:34:31.072 UTC [12259] LOG: listening on Unix socket "/tmp/.s.PGSQL.38243"
2026-09-27 18:34:31.094 UTC [12262] LOG: database system was shut down at 2026-09-27 18:34:29 UTC
2026-09-27 18:34:31.098 UTC [12259] LOG: database system is ready to accept connections
I20260927 18:34:32.089829 389 mini_postgres.cc:96] Postgres bound to 38243
2026-09-27 18:34:32.106 UTC [12270] FATAL: database "slave" does not exist
127.0.97.84:38243 - accepting connections
I20260927 18:34:32.107532 389 mini_ranger.cc:161] Starting Ranger...
I20260927 18:34:32.138571 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:34:32.655371 389 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:34:32.655488 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin
2026-09-27 18:34:32,960 [I] DB FLAVOR :POSTGRES
2026-09-27 18:34:32,961 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:34:32,961 [I] Checking connection..
2026-09-27 18:34:32,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:34:33,203 [I] Checking connection passed.
2026-09-27 18:34:33,203 [I] --------- Verifying version history table ---------
2026-09-27 18:34:33,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:34:33,465 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:34:33,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:34:33,695 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:34:33,696 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:34:33,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:34:33.922 UTC [12334] WARNING: there is no transaction in progress
2026-09-27 18:34:33,929 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:34:33,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:34:34,192 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:34:34,192 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:34:34,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:34:34,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:34:34,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3p4.c.gcp-upstream.internal','N') ;"
2026-09-27 18:34:34,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:34:35,186 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:34:35,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:34:35,445 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:34:35,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:34:35,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:34:35,930 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:34:35,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:34:36.879 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:36.897 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:36.913 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:36.930 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:36.946 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:36.963 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.018 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.025 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.040 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.050 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.060 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.073 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.083 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.090 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.097 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37.103 UTC [12442] WARNING: there is no transaction in progress
2026-09-27 18:34:37,738 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:34:37,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:34:37,959 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:34:37,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:34:38,212 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:34:38,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:34:38,461 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:34:38,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:34:38,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:34:38,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3p4.c.gcp-upstream.internal';"
2026-09-27 18:34:39,220 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:34:39,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:34:39,466 [I] DB_PATCHES have already been applied
I20260927 18:34:39.472014 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKYOLso/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:34:39.472126 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260927 18:34:39.486229 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:34371
Sep 27, 2026 6:34:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:34:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:34:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:34:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:34:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:34:40.625902 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.518s user 0.000s sys 0.041s
Sep 27, 2026 6:35:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:35:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:35:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:35:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:35:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:35:02.389449 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:35:02 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:35:02.560765 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:35:02.580981 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:35:03.447482 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:35:03.447597 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms
I20260927 18:35:03.447656 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:38243
2026-09-27 18:35:03,786 [I] DB FLAVOR :POSTGRES
2026-09-27 18:35:03,787 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:35:03,787 [I] Checking connection
2026-09-27 18:35:03,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:35:04,033 [I] connection success
2026-09-27 18:35:04,033 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:35:04,034 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:35:04,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:35:04,291 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:35:04,291 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:35:04,291 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:35:04,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:38243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:35:04,575 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:35:05.081696 389 mini_ranger_kms.cc:310] Created kms service
I20260927 18:35:05.299103 389 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:35:05.347666 389 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:35:05 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534102, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:05.759213 389 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:35:05.759573 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKYOLso/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKYOLso/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:35:05.759632 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260927 18:35:05.776569 389 mini_ranger_kms.cc:276] Ranger KMS PID: 12622
I20260927 18:35:05.776706 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:47341
Sep 27, 2026 6:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:35:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:35:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:35:08 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:08.104733 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.544s user 0.065s sys 0.291s
I20260927 18:35:08.104981 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:35:08.105180 389 mini_ranger_kms.cc:401] 127.0.97.84:47341/kms/v1/keys
Sep 27 18:35:08 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:35:08 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:35:11 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:35:11.166289 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45453
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45453
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:11.496868 12673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:11.497143 12673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:11.497229 12673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:11.504418 12673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:11.504513 12673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:11.504557 12673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:11.504613 12673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:11.504659 12673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:11.514493 12673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:47341/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45453
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:11.515780 12673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:11.517100 12673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:11.529114 12678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:11.529230 12679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:11.529230 12681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:11.529546 12673 server_base.cc:1034] running on GCE node
I20260927 18:35:11.530632 12673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:11.532006 12673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:11.533249 12673 hybrid_clock.cc:648] HybridClock initialized: now 1790534111533188 us; error 49 us; skew 500 ppm
I20260927 18:35:11.533588 12673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:11 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:11.536684 12673 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:11.538201 12673 webserver.cc:460] Webserver started at http://127.0.97.126:33049/ using document root <none> and password file <none>
I20260927 18:35:11.538846 12673 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:11.538987 12673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:11.539292 12673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:11 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:11.594381 12673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "816992ed1418436ea02d9406dac612e8"
format_stamp: "Formatted at 2026-09-27 18:35:11 on dist-test-slave-n3p4"
server_key: "32a81284bd5ad09260cac9342848abf1"
server_key_iv: "b0ec8e4fe0c74dd9c4bfa52633779e09"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:11.595261 12673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "816992ed1418436ea02d9406dac612e8"
format_stamp: "Formatted at 2026-09-27 18:35:11 on dist-test-slave-n3p4"
server_key: "32a81284bd5ad09260cac9342848abf1"
server_key_iv: "b0ec8e4fe0c74dd9c4bfa52633779e09"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:11.601001 12673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:35:11.621279 12688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:11.622943 12673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:35:11.623136 12673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "816992ed1418436ea02d9406dac612e8"
format_stamp: "Formatted at 2026-09-27 18:35:11 on dist-test-slave-n3p4"
server_key: "32a81284bd5ad09260cac9342848abf1"
server_key_iv: "b0ec8e4fe0c74dd9c4bfa52633779e09"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:11.623359 12673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:11.653512 12673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:11.657822 12673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:11.658217 12673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:11.676978 12673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45453
I20260927 18:35:11.677012 12739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45453 every 8 connection(s)
I20260927 18:35:11.678359 12673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:11.679709 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 12673
I20260927 18:35:11.679917 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:35:11.680752 389 external_mini_cluster.cc:1428] Setting key 188238ae9770fab84ae0e31e026281db
I20260927 18:35:11.683327 12740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:35:11 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:11.696341 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Bootstrap starting.
I20260927 18:35:11.699455 12740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:11.700765 12740 log.cc:824] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:11.703665 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:11.682863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.0.97.126:45453)
0927 18:35:11.683786 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:11.683798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:11.683836 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:35:11.685422 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:11.685444 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:11.685920 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:11.686266 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:11.687922 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:11.689282 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:11.691200 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:11.691846 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:11.691969 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:11.696497 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:11.696507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:11.696528 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:11.696575 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:11.699560 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:11.700407 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:11.700413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:11.700419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:11.700568 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:11.701004 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:11.701008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:11.701010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:11.701695 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:11.702062 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:11.702348 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:35:11.702713 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":521,"threads_started":1}
I20260927 18:35:11.703848 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: No bootstrap required, opened a new log
I20260927 18:35:11.711054 12740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:11.711408 12740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:11.711510 12740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816992ed1418436ea02d9406dac612e8, State: Initialized, Role: FOLLOWER
I20260927 18:35:11.712054 12740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:11.712309 12740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:11.712464 12740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:11.712636 12740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:11.714787 12740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:11.715304 12740 leader_election.cc:302] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816992ed1418436ea02d9406dac612e8; no voters:
I20260927 18:35:11.715883 12740 leader_election.cc:288] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:11.716009 12745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:11.716740 12745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 LEADER]: Becoming Leader. State: Replica: 816992ed1418436ea02d9406dac612e8, State: Running, Role: LEADER
I20260927 18:35:11.717283 12745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:11.718003 12740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:11.719816 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816992ed1418436ea02d9406dac612e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } } }
I20260927 18:35:11.720240 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:11.722891 12751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:11.722757 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816992ed1418436ea02d9406dac612e8. Latest consensus state: current_term: 1 leader_uuid: "816992ed1418436ea02d9406dac612e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } } }
I20260927 18:35:11.723132 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:11.727912 12751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:11.732193 12740 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:35:11.744170 12751 catalog_manager.cc:1347] Generated new cluster ID: b0cb5cac870e4c87b5b3c2a218d2b8ed
I20260927 18:35:11.744406 12751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:11.766822 12751 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:11.768800 12751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:11.786767 12751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Generated new TSK 0
I20260927 18:35:11.787935 12751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:35:12.513221 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:12 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:35:12.940158 12785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:12.940663 12785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:12.940886 12785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:12.953812 12785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:12.953959 12785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:12.954224 12785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:12.971745 12785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:12.973462 12785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:12.975358 12785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:13.007757 12798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:13.008093 12795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:13.008744 12785 server_base.cc:1034] running on GCE node
W20260927 18:35:13.010037 12796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:13.010895 12785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:13.011961 12785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:13.013505 12785 hybrid_clock.cc:648] HybridClock initialized: now 1790534113013446 us; error 55 us; skew 500 ppm
I20260927 18:35:13.013907 12785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:13 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:13.017694 12785 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:13.023169 12785 webserver.cc:460] Webserver started at http://127.0.97.65:39107/ using document root <none> and password file <none>
I20260927 18:35:13.023883 12785 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:13.024031 12785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:13.024358 12785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:13 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:13.043143 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "07c5dca3afde407b8ca10a8ad1a136b8"
format_stamp: "Formatted at 2026-09-27 18:35:13 on dist-test-slave-n3p4"
server_key: "8390475bb0d0784fdae5cfbd3085a1ec"
server_key_iv: "9da56e0eb0837edb91dcc16bd59a9b4d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:13.044138 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "07c5dca3afde407b8ca10a8ad1a136b8"
format_stamp: "Formatted at 2026-09-27 18:35:13 on dist-test-slave-n3p4"
server_key: "8390475bb0d0784fdae5cfbd3085a1ec"
server_key_iv: "9da56e0eb0837edb91dcc16bd59a9b4d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:13.050814 12785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:35:13.085292 12805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:13.087242 12785 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.006s sys 0.009s
I20260927 18:35:13.087476 12785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "07c5dca3afde407b8ca10a8ad1a136b8"
format_stamp: "Formatted at 2026-09-27 18:35:13 on dist-test-slave-n3p4"
server_key: "8390475bb0d0784fdae5cfbd3085a1ec"
server_key_iv: "9da56e0eb0837edb91dcc16bd59a9b4d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:13.087841 12785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:13.151146 12785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:13.158520 12785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:13.158938 12785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:13.166275 12785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:13.174407 12785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:13.174563 12785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:13.174760 12785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:13.174926 12785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:35:13 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:13.394979 12785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41101
I20260927 18:35:13.395004 12917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41101 every 8 connection(s)
I20260927 18:35:13.396607 12785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:13.403498 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 12785
I20260927 18:35:13.403792 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:13.404814 389 external_mini_cluster.cc:1428] Setting key a9ba6d719afa5265f0cfe5971aaf8bc6
Sep 27 18:35:13 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:35:13.460034 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:13.400256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.65:36745)
0927 18:35:13.426940 (+ 26684us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:13.427010 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:13.436650 (+ 9640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:13.438807 (+ 2157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:13.438820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:13.439462 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:13.440333 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:13.440367 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:13.441995 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:13.442000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:13.444249 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:13.444274 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:13.445503 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:13.445510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:13.446952 (+ 1442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:13.447860 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:13.447889 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:13.450776 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:13.455169 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:13.455179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:13.455197 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:13.455807 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:13.456244 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:13.456248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:13.456252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:13.456530 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:13.457269 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:13.457277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:13.457719 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 18:35:13.458037 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 18:35:13.458583 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26358,"thread_start_us":24783,"threads_started":1}
I20260927 18:35:13.461218 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:13.399483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36745 (local address 127.0.97.126:45453)
0927 18:35:13.437459 (+ 37976us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:13.437465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:13.437486 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:35:13.437546 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:13.437551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:13.437653 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:13.437768 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:13.440589 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:13.441761 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:13.444451 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:13.445074 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:13.445192 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:13.451115 (+ 5923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:13.451122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:13.451127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:13.451170 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:13.454957 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:13.455955 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:13.455961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:13.455965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:13.456110 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:13.456676 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:13.456681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:13.456684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:13.457049 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:13.457190 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:13.460620 (+ 3430us) server_negotiation.cc:299] Negotiation successful
0927 18:35:13.460844 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37841,"thread_start_us":42326,"threads_started":1}
I20260927 18:35:13.463524 12918 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:35:13.463938 12918 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:13.464823 12918 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:13.467329 12705 ts_manager.cc:194] Registered new tserver with Master: 07c5dca3afde407b8ca10a8ad1a136b8 (127.0.97.65:41101)
I20260927 18:35:13.469229 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:36745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:35:13.499116 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:13 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260927 18:35:13.973928 12924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:13.974267 12924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:13.974439 12924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:13.982616 12924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:13.982760 12924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:13.982942 12924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:13.992764 12924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:13.995090 12924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:13.996450 12924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:14.011310 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:14.011931 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:14.011953 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:14.012411 12924 server_base.cc:1034] running on GCE node
I20260927 18:35:14.013068 12924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:14.014230 12924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:14.015479 12924 hybrid_clock.cc:648] HybridClock initialized: now 1790534114015425 us; error 46 us; skew 500 ppm
I20260927 18:35:14.015828 12924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:14.022498 12924 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:14.024101 12924 webserver.cc:460] Webserver started at http://127.0.97.66:46849/ using document root <none> and password file <none>
I20260927 18:35:14.024838 12924 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:14.024989 12924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:14.025295 12924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:14.043896 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "28c8a3109c8a4308a85388a30132ceaa"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "e04724f3334b8739dbd9161fd28843d1"
server_key_iv: "c192aa58306cbe2077fc41aee3ff4d64"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.045056 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "28c8a3109c8a4308a85388a30132ceaa"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "e04724f3334b8739dbd9161fd28843d1"
server_key_iv: "c192aa58306cbe2077fc41aee3ff4d64"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.052652 12924 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 18:35:14.071796 12948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:14.073768 12924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:35:14.074092 12924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "28c8a3109c8a4308a85388a30132ceaa"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "e04724f3334b8739dbd9161fd28843d1"
server_key_iv: "c192aa58306cbe2077fc41aee3ff4d64"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.074445 12924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:14.115985 12924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:14.121712 12924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:14.122169 12924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:14.123404 12924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:14.125398 12924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:14.125526 12924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:14.125684 12924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:14.125763 12924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:14.172509 12924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41119
I20260927 18:35:14.172552 13060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41119 every 8 connection(s)
I20260927 18:35:14.174314 12924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:14.180662 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 12924
I20260927 18:35:14.180845 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:14.181409 389 external_mini_cluster.cc:1428] Setting key ca6d0ed91961ad13f1f33c35f8a269fb
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260927 18:35:14.219897 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.177927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.66:51601)
0927 18:35:14.186490 (+ 8563us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:14.186552 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:14.189379 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:14.200125 (+ 10746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:14.200145 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:14.200855 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:14.201772 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:14.201823 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.203568 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.203577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:14.205511 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:14.205531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.206433 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.206441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:14.206801 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.207704 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:14.207744 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:14.210791 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:14.214689 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:14.214700 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:14.214721 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:14.215350 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:14.215824 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:14.215831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:14.215837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:14.216104 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:14.216859 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:14.216870 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:14.217347 (+ 477us) client_negotiation.cc:769] Sending connection context
0927 18:35:14.217701 (+ 354us) client_negotiation.cc:240] Negotiation successful
0927 18:35:14.218322 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8229,"thread_start_us":7910,"threads_started":1}
I20260927 18:35:14.221383 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.178531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51601 (local address 127.0.97.126:45453)
0927 18:35:14.199361 (+ 20830us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:14.199367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:14.199392 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:35:14.199461 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:14.199466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:14.199563 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:14.199716 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:14.202048 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.203358 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.205700 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.206188 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.206332 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.211087 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:14.211096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:14.211103 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:14.211154 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:14.214499 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.215506 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.215515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.215520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.215682 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.216238 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.216245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.216250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.216646 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:14.216778 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:14.220640 (+ 3862us) server_negotiation.cc:299] Negotiation successful
0927 18:35:14.220909 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20670,"thread_start_us":21281,"threads_started":1}
I20260927 18:35:14.222683 13061 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
I20260927 18:35:14.223031 13061 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:14.223951 13061 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:14.226405 12705 ts_manager.cc:194] Registered new tserver with Master: 28c8a3109c8a4308a85388a30132ceaa (127.0.97.66:41119)
I20260927 18:35:14.227773 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:35:14.248128 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:14.472878 12918 heartbeater.cc:503] Master 127.0.97.126:45453 was elected leader, sending a full tablet report...
W20260927 18:35:14.575459 13067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:14.575740 13067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:14.575891 13067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:14.583513 13067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:14.583627 13067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:14.583829 13067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:14.593676 13067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:14.595247 13067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:14.596669 13067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:14.611487 13072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:14.611413 13073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:14.612172 13067 server_base.cc:1034] running on GCE node
W20260927 18:35:14.612228 13075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:14.612864 13067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:14.613878 13067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:14.615108 13067 hybrid_clock.cc:648] HybridClock initialized: now 1790534114615049 us; error 44 us; skew 500 ppm
I20260927 18:35:14.615432 13067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:14.618664 13067 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:14.620301 13067 webserver.cc:460] Webserver started at http://127.0.97.67:40833/ using document root <none> and password file <none>
I20260927 18:35:14.620967 13067 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:14.621119 13067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:14.621419 13067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:14.636391 13067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "f979ca9b58e1474e802d8111ecca17d2"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "ab78f6cdec2f1d85824411b6d2e26aa7"
server_key_iv: "777dc56b9b2cfd2ae765494b3624ae21"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.637281 13067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "f979ca9b58e1474e802d8111ecca17d2"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "ab78f6cdec2f1d85824411b6d2e26aa7"
server_key_iv: "777dc56b9b2cfd2ae765494b3624ae21"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.642748 13067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 18:35:14.658855 13082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:14.660481 13067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:35:14.660651 13067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "f979ca9b58e1474e802d8111ecca17d2"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "ab78f6cdec2f1d85824411b6d2e26aa7"
server_key_iv: "777dc56b9b2cfd2ae765494b3624ae21"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:14.660888 13067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:14.673242 13067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:14.677410 13067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:14.677711 13067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:14.678766 13067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:14.680408 13067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:14.680562 13067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:14.680701 13067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:14.680776 13067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:14.719815 13067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35055
I20260927 18:35:14.719915 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35055 every 8 connection(s)
I20260927 18:35:14.721124 13067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:14.731492 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13067
I20260927 18:35:14.731684 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:14.732417 389 external_mini_cluster.cc:1428] Setting key 8152dce7c60537afa86e3b9cf8c8408d
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:14.741536 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.723880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55371 (local address 127.0.97.126:45453)
0927 18:35:14.724100 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:14.724107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:14.725291 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 18:35:14.727236 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:14.727242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:14.727389 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:14.727521 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:14.729017 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.730220 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.731893 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.732614 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.732764 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.736333 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:14.736338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:14.736341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:14.736374 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:14.739015 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.739660 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.739664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.739666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.739818 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.740178 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.740181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.740183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.740455 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:14.740563 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:14.741152 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 18:35:14.741310 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:35:14.742584 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.724290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.67:55371)
0927 18:35:14.725111 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:14.725161 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:14.726968 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:14.727739 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:14.727748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:14.728238 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:14.728849 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:14.728865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.730377 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.730381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:14.731710 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:14.731736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.732819 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.732824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:14.733070 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.733848 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:14.733881 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:14.736126 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:14.739136 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:14.739143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:14.739156 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:14.739561 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:14.739918 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:14.739921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:14.739924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:14.740088 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:14.740553 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:14.740559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:14.740865 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 18:35:14.741082 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 18:35:14.741436 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":368,"threads_started":1}
I20260927 18:35:14.743997 13195 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
I20260927 18:35:14.744292 13195 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:14.744972 13195 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:14.746516 12705 ts_manager.cc:194] Registered new tserver with Master: f979ca9b58e1474e802d8111ecca17d2 (127.0.97.67:35055)
I20260927 18:35:14.747354 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:55371
I20260927 18:35:14.748440 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:14.750162 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 12785
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:35:14.765609 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.751029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.65:41101)
0927 18:35:14.751600 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:14.751606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:14.751631 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:35:14.751791 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:14.751802 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:14.752052 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:14.752177 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:14.753556 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.754599 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.755906 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.756419 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.756518 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.760386 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:14.760397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:14.760409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:14.760440 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:14.763010 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.763623 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.763645 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.763647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.763785 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.764138 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.764146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.764149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.764525 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:14.764756 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:14.764953 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:35:14.765199 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":282,"threads_started":1}
I20260927 18:35:14.766800 12913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:14.849088 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 12785
I20260927 18:35:14.864920 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 12924
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:35:14.880355 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.865904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.97.66:41119)
0927 18:35:14.866424 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:14.866431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:14.866455 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:35:14.866527 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:14.866533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:14.866738 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:14.866871 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:14.868118 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.869098 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.870505 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.871050 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.871145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.875135 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:14.875146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:14.875160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:14.875190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:14.877758 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.878567 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.878574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.878576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.878707 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.878993 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.878996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.878998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.879349 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:14.879629 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:14.879796 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 18:35:14.880082 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":266,"threads_started":1}
I20260927 18:35:14.881630 13056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:14.963410 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 12924
I20260927 18:35:14.980643 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13067
Sep 27 18:35:14 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:35:14.997190 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:14.981749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.0.97.67:35055)
0927 18:35:14.982452 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:14.982459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:14.982490 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:35:14.982550 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:14.982557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:14.982812 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:14.982962 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:14.984260 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.985351 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.986835 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:14.987428 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:14.987536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:14.991921 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:14.991931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:14.991946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:14.991979 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:14.994709 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.995377 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.995384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.995386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.995500 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:14.995892 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:14.995895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:14.995897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:14.996216 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:14.996440 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:14.996701 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:35:14.996881 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":365,"threads_started":1}
I20260927 18:35:14.998633 13190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:15.081735 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13067
I20260927 18:35:15.097117 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 12673
I20260927 18:35:15.098518 12735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:15.205029 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 12673
I20260927 18:35:15.226244 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45453
--webserver_interface=127.0.97.126
--webserver_port=33049
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45453
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:15.548638 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:15.549017 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:15.549196 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:15.556371 13206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:15.556468 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:15.556581 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:15.556672 13206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:15.556746 13206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:15.566617 13206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45453
--ranger_config_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=33049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:15.567858 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:15.569146 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:15.581049 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:15.581023 13214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:15.581077 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:15.581980 13206 server_base.cc:1034] running on GCE node
I20260927 18:35:15.582672 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:15.583977 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:15.585179 13206 hybrid_clock.cc:648] HybridClock initialized: now 1790534115585132 us; error 37 us; skew 500 ppm
I20260927 18:35:15.585537 13206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:15 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:15.588735 13206 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:15.590303 13206 webserver.cc:460] Webserver started at http://127.0.97.126:33049/ using document root <none> and password file <none>
I20260927 18:35:15.590924 13206 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:15.591056 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:35:15 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:15.659502 13206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:35:15.663266 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:15.664882 13206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:35:15.665103 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "816992ed1418436ea02d9406dac612e8"
format_stamp: "Formatted at 2026-09-27 18:35:11 on dist-test-slave-n3p4"
server_key: "32a81284bd5ad09260cac9342848abf1"
server_key_iv: "b0ec8e4fe0c74dd9c4bfa52633779e09"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:15.666147 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:15.689188 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:15.693133 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:15.693418 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:15.712978 13206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45453
I20260927 18:35:15.712998 13273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45453 every 8 connection(s)
I20260927 18:35:15.714581 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:15.721746 13274 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:35:15.722280 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13206
I20260927 18:35:15.723143 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:41101
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:45453
--webserver_port=39107
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:15.725503 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Bootstrap starting.
I20260927 18:35:15.745702 13274 log.cc:824] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:15.755666 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:35:15.756271 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Bootstrap complete.
I20260927 18:35:15.765952 13274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:15.766870 13274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816992ed1418436ea02d9406dac612e8, State: Initialized, Role: FOLLOWER
I20260927 18:35:15.767467 13274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:15.767719 13274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:15.767858 13274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:15.768042 13274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:35:15.772356 13274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:15.772854 13274 leader_election.cc:302] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816992ed1418436ea02d9406dac612e8; no voters:
I20260927 18:35:15.773535 13274 leader_election.cc:288] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:35:15.773695 13278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:35:15.776263 13274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:15.780853 13278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [term 2 LEADER]: Becoming Leader. State: Replica: 816992ed1418436ea02d9406dac612e8, State: Running, Role: LEADER
I20260927 18:35:15.781462 13278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } }
I20260927 18:35:15.784721 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "816992ed1418436ea02d9406dac612e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } } }
I20260927 18:35:15.785073 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:15.785688 13274 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:35:15.785567 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816992ed1418436ea02d9406dac612e8. Latest consensus state: current_term: 2 leader_uuid: "816992ed1418436ea02d9406dac612e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816992ed1418436ea02d9406dac612e8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45453 } } }
I20260927 18:35:15.785940 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:15.806463 13293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:15.813825 13293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:15.816190 13293 catalog_manager.cc:1259] Loaded cluster ID: b0cb5cac870e4c87b5b3c2a218d2b8ed
I20260927 18:35:15.816490 13293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:15.821895 13293 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:15.823945 13293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Loaded TSK: 0
I20260927 18:35:15.825887 13293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:35:16.164667 13276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:16.165091 13276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:16.165315 13276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:16.177474 13276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:16.177621 13276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:16.177912 13276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:16.193742 13276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:41101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=39107
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:16.195569 13276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:16.197348 13276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:16.247291 13311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:16.255633 13312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:16.256393 13276 server_base.cc:1034] running on GCE node
W20260927 18:35:16.257220 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:16.258176 13276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:16.264118 13276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:16.265396 13276 hybrid_clock.cc:648] HybridClock initialized: now 1790534116265326 us; error 63 us; skew 500 ppm
I20260927 18:35:16.265868 13276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:16 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:16.270921 13276 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:16.305627 13276 webserver.cc:460] Webserver started at http://127.0.97.65:39107/ using document root <none> and password file <none>
I20260927 18:35:16.306846 13276 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:16.307108 13276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:35:16 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:16.372203 13276 fs_manager.cc:709] Time spent opening directory manager: real 0.024s user 0.010s sys 0.001s
I20260927 18:35:16.384153 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:16.386106 13276 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.004s
I20260927 18:35:16.386344 13276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "07c5dca3afde407b8ca10a8ad1a136b8"
format_stamp: "Formatted at 2026-09-27 18:35:13 on dist-test-slave-n3p4"
server_key: "8390475bb0d0784fdae5cfbd3085a1ec"
server_key_iv: "9da56e0eb0837edb91dcc16bd59a9b4d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:16.387894 13276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:16.424466 13276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:16.429387 13276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:16.429664 13276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:16.430742 13276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:16.432248 13276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:16.432377 13276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:16.432502 13276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:16.432585 13276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:16.515065 13276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41101
I20260927 18:35:16.515095 13435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41101 every 8 connection(s)
I20260927 18:35:16.516963 13276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:16.520396 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13276
I20260927 18:35:16.521203 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:41119
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:45453
--webserver_port=46849
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:16 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:16.558717 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:16.520210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.65:41047)
0927 18:35:16.531826 (+ 11616us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:16.531884 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:16.534438 (+ 2554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:16.536970 (+ 2532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:16.536981 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:16.537630 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:16.538481 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:16.538514 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:16.540941 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:16.540947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:16.543132 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:16.543148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:16.544949 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:16.544957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:16.545245 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:16.546136 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:16.546169 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:16.548919 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:16.553556 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:16.553567 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:16.553585 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:16.554205 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:16.554738 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:16.554743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:16.554746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:16.554961 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:16.555958 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:16.555967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:16.556367 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 18:35:16.556652 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 18:35:16.557267 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11244,"thread_start_us":11575,"threads_started":1}
I20260927 18:35:16.561178 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:16.520205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41047 (local address 127.0.97.126:45453)
0927 18:35:16.533648 (+ 13443us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:16.533669 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:16.533720 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:35:16.535322 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:16.535344 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:16.535838 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:16.536266 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:16.538743 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:16.540739 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:16.543332 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:16.544372 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:16.544560 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:16.549267 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:16.549278 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:16.549300 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:16.549355 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:16.553397 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:16.554391 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:16.554401 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:16.554408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:16.554611 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:16.555104 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:16.555110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:16.555114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:16.555818 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:16.556008 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:16.559439 (+ 3431us) server_negotiation.cc:299] Negotiation successful
0927 18:35:16.559860 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13164,"thread_start_us":13459,"threads_started":1}
I20260927 18:35:16.564776 13436 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
I20260927 18:35:16.565150 13436 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:16.566082 13436 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
W20260927 18:35:16.569362 13436 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:45453 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:35:16.569659 13436 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:16.570009 13436 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:16.571262 13436 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:16.571565 13436 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
W20260927 18:35:16.572681 13436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:35:16.941243 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:16.941650 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:16.941880 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:16.953444 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:16.953646 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:16.953883 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:16.969946 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:41119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=46849
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:16.971710 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:16.973441 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:16.990808 13456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:16.990772 13457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:16.998107 13459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:16.998692 13440 server_base.cc:1034] running on GCE node
I20260927 18:35:16.999452 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:17.000669 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:17.001982 13440 hybrid_clock.cc:648] HybridClock initialized: now 1790534117001904 us; error 55 us; skew 500 ppm
I20260927 18:35:17.002353 13440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:16 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:17.006559 13440 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:17.008509 13440 webserver.cc:460] Webserver started at http://127.0.97.66:46849/ using document root <none> and password file <none>
I20260927 18:35:17.009513 13440 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:17.009701 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:17.034734 13440 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:35:17.054503 13467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:17.057125 13440 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.008s sys 0.012s
I20260927 18:35:17.057406 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "28c8a3109c8a4308a85388a30132ceaa"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "e04724f3334b8739dbd9161fd28843d1"
server_key_iv: "c192aa58306cbe2077fc41aee3ff4d64"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:17.059155 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:17.080226 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:17.084546 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:17.084843 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:17.085942 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:17.087433 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:17.087553 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:17.087694 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:17.087782 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:17.140616 13440 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41119
I20260927 18:35:17.140674 13580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41119 every 8 connection(s)
I20260927 18:35:17.142591 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:17.144505 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13440
I20260927 18:35:17.145279 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:47341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:35055
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:45453
--webserver_port=40833
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:17.199766 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:17.148365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:47469 (local address 127.0.97.126:45453)
0927 18:35:17.169097 (+ 20732us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:17.169108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:17.170386 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 18:35:17.177964 (+ 7578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:17.177974 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:17.178076 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:17.178161 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:17.180534 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:17.181802 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:17.185267 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:17.186060 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:17.186171 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:17.191425 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:17.191435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:17.191444 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:17.191503 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:17.195652 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:17.196757 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:17.196764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:17.196769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:17.196935 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:17.197519 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:17.197526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:17.197532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:17.197980 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:17.198099 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:17.199111 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0927 18:35:17.199339 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20507,"thread_start_us":20758,"threads_started":1}
I20260927 18:35:17.202414 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:17.147593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.66:47469)
0927 18:35:17.170029 (+ 22436us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:17.170169 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:17.177570 (+ 7401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:17.178721 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:17.178735 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:17.179396 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:17.180326 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:17.180358 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:17.182000 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:17.182007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:17.185099 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:17.185117 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:17.186509 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:17.186518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:17.186875 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:17.187815 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:17.187847 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:17.191041 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:17.195876 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:17.195889 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:17.195912 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:17.196602 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:17.197101 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:17.197108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:17.197114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:17.197373 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:17.198208 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:17.198217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:17.198651 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 18:35:17.200314 (+ 1663us) client_negotiation.cc:240] Negotiation successful
0927 18:35:17.200961 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21966,"thread_start_us":26719,"threads_started":1}
I20260927 18:35:17.204432 13581 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
I20260927 18:35:17.204771 13581 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:17.205698 13581 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:17.207697 13239 ts_manager.cc:194] Registered new tserver with Master: 28c8a3109c8a4308a85388a30132ceaa (127.0.97.66:41119)
I20260927 18:35:17.209878 13239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:47469
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:17.572821 13436 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:17.573277 13436 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:17.574697 13239 ts_manager.cc:194] Registered new tserver with Master: 07c5dca3afde407b8ca10a8ad1a136b8 (127.0.97.65:41101)
I20260927 18:35:17.575968 13239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:41047
W20260927 18:35:17.651901 13585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:17.652453 13585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:17.652696 13585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:17.666693 13585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:17.666831 13585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:17.667042 13585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:17.681864 13585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:47341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:35055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=40833
--tserver_master_addrs=127.0.97.126:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:17.683272 13585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:17.684645 13585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:17.700105 13591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:17.701232 13585 server_base.cc:1034] running on GCE node
W20260927 18:35:17.701671 13590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:17.702080 13593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:17.702812 13585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:17.703943 13585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:17.705222 13585 hybrid_clock.cc:648] HybridClock initialized: now 1790534117705170 us; error 32 us; skew 500 ppm
I20260927 18:35:17.705597 13585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:17.709625 13585 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:17.723975 13585 webserver.cc:460] Webserver started at http://127.0.97.67:40833/ using document root <none> and password file <none>
I20260927 18:35:17.724941 13585 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:17.725100 13585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:17.756374 13585 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 18:35:17.778477 13601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:17.780560 13585 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.015s
I20260927 18:35:17.780817 13585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "f979ca9b58e1474e802d8111ecca17d2"
format_stamp: "Formatted at 2026-09-27 18:35:14 on dist-test-slave-n3p4"
server_key: "ab78f6cdec2f1d85824411b6d2e26aa7"
server_key_iv: "777dc56b9b2cfd2ae765494b3624ae21"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:17.782380 13585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:17.858076 13585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:17.863461 13585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:17.863849 13585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:17.866189 13585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:17.868011 13585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:17.868160 13585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 18:35:17.868294 13585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:17.868394 13585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:35:17 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260927 18:35:18.018869 13585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:35055
I20260927 18:35:18.018949 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:35055 every 8 connection(s)
I20260927 18:35:18.020787 13585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:18.025759 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13585
Sep 27 18:35:18 dist-test-slave-n3p4 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:18.074898 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.025318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45453 (local address 127.0.97.67:43769)
0927 18:35:18.037166 (+ 11848us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.037220 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.045003 (+ 7783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.052325 (+ 7322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.052340 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.053001 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:18.053938 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.053968 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.055671 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.055677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.057729 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.057745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.060003 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.060011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.060316 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.061189 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:18.061226 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:18.063934 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:18.069608 (+ 5674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:18.069620 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:18.069643 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:18.070375 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:18.070904 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:18.070911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:18.070915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:18.071201 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:18.071780 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:18.071788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:18.072277 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.072700 (+ 423us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.073319 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11498,"thread_start_us":13691,"threads_started":1}
I20260927 18:35:18.076821 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.024758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43769 (local address 127.0.97.126:45453)
0927 18:35:18.051540 (+ 26782us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.051547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.051572 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:35:18.051644 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.051650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.051747 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.054171 (+ 2424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.054338 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.055491 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.058017 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.058685 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.060767 (+ 2082us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.064233 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.064240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.064245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.064296 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.069408 (+ 5112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.070586 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.070591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.070595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.070745 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.071340 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.071344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.071349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.071637 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.071808 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.076112 (+ 4304us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.076367 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26635,"thread_start_us":34906,"threads_started":1}
I20260927 18:35:18.078598 13715 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45453
I20260927 18:35:18.078998 13715 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:18.079945 13715 heartbeater.cc:511] Master 127.0.97.126:45453 requested a full tablet report, sending...
I20260927 18:35:18.081888 13239 ts_manager.cc:194] Registered new tserver with Master: f979ca9b58e1474e802d8111ecca17d2 (127.0.97.67:35055)
I20260927 18:35:18.082581 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.026764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.0.97.126:45453)
0927 18:35:18.059140 (+ 32376us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.059146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.060023 (+ 877us) server_negotiation.cc:407] Connection header received
0927 18:35:18.060078 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.060094 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.060188 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.060289 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.062610 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.063814 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.066018 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.066709 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.066837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.071216 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.071224 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.071229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.071271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.079137 (+ 7866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.080178 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.080185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.080188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.080353 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.080879 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.080884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.080888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.081254 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.081578 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.081947 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.082191 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32244,"thread_start_us":25116,"threads_started":1}
I20260927 18:35:18.083067 13239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:43769
I20260927 18:35:18.086006 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:18.116492 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.094175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.97.126:45453)
0927 18:35:18.094390 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.094395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.102249 (+ 7854us) server_negotiation.cc:407] Connection header received
0927 18:35:18.102549 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.102555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.102637 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.102744 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.104256 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.104958 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.106644 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.107261 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.107356 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.110877 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.110883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.110888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.110928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.113542 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.114435 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.114441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.114444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.114583 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.115028 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.115033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.115043 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.115304 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.115583 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.115916 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.116159 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:35:18.122785 13239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:18.130332 13239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:18.179015 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.145012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:35055 (local address 127.0.0.1:52734)
0927 18:35:18.147652 (+ 2640us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.147702 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.147880 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.159532 (+ 11652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.159543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.159810 (+ 267us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:18.164126 (+ 4316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.164149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.167067 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.167073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.169464 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.169484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.177136 (+ 7652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.177145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.177360 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.177652 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.178486 (+ 834us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.178676 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2366,"thread_start_us":2234,"threads_started":1}
I20260927 18:35:18.182266 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.145691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.0.97.67:35055)
0927 18:35:18.147072 (+ 1381us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.147090 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.148134 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 18:35:18.148238 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.148245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.148494 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.148675 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:18.164449 (+ 15774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.166898 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.170283 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.172413 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.172588 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.172915 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.181664 (+ 8749us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.181896 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":1061,"threads_started":1}
I20260927 18:35:18.187139 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.148895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41119 (local address 127.0.0.1:47130)
0927 18:35:18.174095 (+ 25200us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.174121 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.174240 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.176439 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.176444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.176471 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:18.176863 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.176875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.181456 (+ 4581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.181460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.183876 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.183891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.186382 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.186389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.186552 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.186585 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.186680 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.186804 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25052,"thread_start_us":29011,"threads_started":1}
I20260927 18:35:18.186733 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 82c559b16f714ef397f7f2316ff070ac (DEFAULT_TABLE table=test-table [id=535b811c46e54b4487ca533dbd8a7f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:18.188760 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c559b16f714ef397f7f2316ff070ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:18.189121 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.149171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.0.97.66:41119)
0927 18:35:18.175333 (+ 26162us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.175349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.175381 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:35:18.175495 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.175504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.175792 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.176007 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:18.179500 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.181277 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.184060 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.185883 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.186013 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.186246 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.188548 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.188711 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25980,"thread_start_us":27037,"threads_started":1}
I20260927 18:35:18.192250 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.152415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41101 (local address 127.0.0.1:43184)
0927 18:35:18.178258 (+ 25843us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.178276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.178377 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.184870 (+ 6493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.184876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.184902 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:18.185172 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.185183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.187092 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.187097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.189286 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.189301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.191461 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.191467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.191671 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.191705 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.191796 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.191927 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25737,"thread_start_us":21394,"threads_started":1}
I20260927 18:35:18.192965 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.153155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.0.97.65:41101)
0927 18:35:18.184021 (+ 30866us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.184030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.184060 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:35:18.184160 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.184168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.184412 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.184620 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:18.185376 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.186935 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.189533 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.190996 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.191113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.191318 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.192477 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.192640 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30689,"thread_start_us":36628,"threads_started":1}
I20260927 18:35:18.192796 13516 tablet_service.cc:1467] Processing CreateTablet for tablet 82c559b16f714ef397f7f2316ff070ac (DEFAULT_TABLE table=test-table [id=535b811c46e54b4487ca533dbd8a7f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:18.194247 13516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c559b16f714ef397f7f2316ff070ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:18.196309 13371 tablet_service.cc:1467] Processing CreateTablet for tablet 82c559b16f714ef397f7f2316ff070ac (DEFAULT_TABLE table=test-table [id=535b811c46e54b4487ca533dbd8a7f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:18.197644 13371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c559b16f714ef397f7f2316ff070ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:18.214375 13581 heartbeater.cc:503] Master 127.0.97.126:45453 was elected leader, sending a full tablet report...
I20260927 18:35:18.216169 13738 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Bootstrap starting.
I20260927 18:35:18.218658 13739 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Bootstrap starting.
I20260927 18:35:18.220352 13738 tablet_bootstrap.cc:654] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:18.221705 13738 log.cc:824] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:18.222855 13739 tablet_bootstrap.cc:654] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:18.224273 13739 log.cc:824] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:18.232406 13738 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: No bootstrap required, opened a new log
I20260927 18:35:18.232847 13738 ts_tablet_manager.cc:1397] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260927 18:35:18.238492 13742 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Bootstrap starting.
I20260927 18:35:18.241983 13742 tablet_bootstrap.cc:654] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:18.243204 13742 log.cc:824] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:18.243781 13739 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: No bootstrap required, opened a new log
I20260927 18:35:18.243726 13738 raft_consensus.cc:348] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.244036 13739 ts_tablet_manager.cc:1397] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.014s
I20260927 18:35:18.244170 13738 raft_consensus.cc:374] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:18.244257 13738 raft_consensus.cc:729] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f979ca9b58e1474e802d8111ecca17d2, State: Initialized, Role: FOLLOWER
I20260927 18:35:18.244966 13738 consensus_queue.cc:260] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.252533 13742 tablet_bootstrap.cc:492] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: No bootstrap required, opened a new log
I20260927 18:35:18.252872 13742 ts_tablet_manager.cc:1397] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260927 18:35:18.258543 13739 raft_consensus.cc:348] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.259246 13739 raft_consensus.cc:374] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:18.259872 13739 raft_consensus.cc:729] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c5dca3afde407b8ca10a8ad1a136b8, State: Initialized, Role: FOLLOWER
I20260927 18:35:18.260789 13738 ts_tablet_manager.cc:1428] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Time spent starting tablet: real 0.028s user 0.018s sys 0.007s
I20260927 18:35:18.261117 13715 heartbeater.cc:503] Master 127.0.97.126:45453 was elected leader, sending a full tablet report...
I20260927 18:35:18.260716 13739 consensus_queue.cc:260] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.263754 13742 raft_consensus.cc:348] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.264220 13742 raft_consensus.cc:374] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:18.264356 13742 raft_consensus.cc:729] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c8a3109c8a4308a85388a30132ceaa, State: Initialized, Role: FOLLOWER
I20260927 18:35:18.265101 13742 consensus_queue.cc:260] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.269905 13436 heartbeater.cc:503] Master 127.0.97.126:45453 was elected leader, sending a full tablet report...
I20260927 18:35:18.270166 13742 ts_tablet_manager.cc:1428] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260927 18:35:18.270677 13739 ts_tablet_manager.cc:1428] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Time spent starting tablet: real 0.027s user 0.020s sys 0.005s
W20260927 18:35:18.296373 13716 tablet.cc:2369] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:35:18.428489 13582 tablet.cc:2369] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:18.507923 13749 raft_consensus.cc:482] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:18.508261 13749 raft_consensus.cc:504] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.510026 13749 leader_election.cc:288] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28c8a3109c8a4308a85388a30132ceaa (127.0.97.66:41119), 07c5dca3afde407b8ca10a8ad1a136b8 (127.0.97.65:41101)
I20260927 18:35:18.517266 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.510885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:37341 (local address 127.0.97.65:41101)
0927 18:35:18.511094 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.511100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.511329 (+ 229us) server_negotiation.cc:407] Connection header received
0927 18:35:18.511478 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.511482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.511549 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.511658 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:18.512174 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.513184 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.515495 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.516327 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.516488 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.516582 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.516880 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.517008 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:35:18.517388 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.510473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41101 (local address 127.0.97.67:37341)
0927 18:35:18.511208 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.511224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.511353 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.511697 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.511700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.511720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:18.511987 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.511994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.513355 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.513362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.515332 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.515343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.516527 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.516533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.516720 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.516745 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.516853 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.516979 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":59,"thread_start_us":393,"threads_started":1}
I20260927 18:35:18.517921 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.510354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50667 (local address 127.0.97.66:41119)
0927 18:35:18.510560 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.510583 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.510695 (+ 112us) server_negotiation.cc:407] Connection header received
0927 18:35:18.510996 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.511004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.511092 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.511175 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:18.511660 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.512679 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.515104 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.516332 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.516461 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.516561 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.517539 (+ 978us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.517671 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:35:18.517442 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.510354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41119 (local address 127.0.97.67:50667)
0927 18:35:18.510553 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:18.510602 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:18.510754 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:18.511235 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:18.511243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:18.511282 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:18.511520 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.511529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.512835 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.512840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.514924 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:18.514939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.516675 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.516681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:18.516882 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.516917 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:35:18.517002 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:35:18.517148 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:35:18.518555 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c559b16f714ef397f7f2316ff070ac" candidate_uuid: "f979ca9b58e1474e802d8111ecca17d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" is_pre_election: true
I20260927 18:35:18.519214 13391 raft_consensus.cc:2454] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f979ca9b58e1474e802d8111ecca17d2 in term 0.
I20260927 18:35:18.519135 13536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c559b16f714ef397f7f2316ff070ac" candidate_uuid: "f979ca9b58e1474e802d8111ecca17d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c8a3109c8a4308a85388a30132ceaa" is_pre_election: true
I20260927 18:35:18.519709 13536 raft_consensus.cc:2454] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f979ca9b58e1474e802d8111ecca17d2 in term 0.
W20260927 18:35:18.519886 13437 tablet.cc:2369] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:18.519923 13602 leader_election.cc:302] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c5dca3afde407b8ca10a8ad1a136b8, f979ca9b58e1474e802d8111ecca17d2; no voters:
I20260927 18:35:18.520398 13749 raft_consensus.cc:2781] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:18.520548 13749 raft_consensus.cc:482] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:18.520653 13749 raft_consensus.cc:3037] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:18.523662 13749 raft_consensus.cc:504] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.524364 13749 leader_election.cc:288] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [CANDIDATE]: Term 1 election: Requested vote from peers 28c8a3109c8a4308a85388a30132ceaa (127.0.97.66:41119), 07c5dca3afde407b8ca10a8ad1a136b8 (127.0.97.65:41101)
I20260927 18:35:18.524792 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c559b16f714ef397f7f2316ff070ac" candidate_uuid: "f979ca9b58e1474e802d8111ecca17d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c5dca3afde407b8ca10a8ad1a136b8"
I20260927 18:35:18.524775 13536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c559b16f714ef397f7f2316ff070ac" candidate_uuid: "f979ca9b58e1474e802d8111ecca17d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c8a3109c8a4308a85388a30132ceaa"
I20260927 18:35:18.524971 13536 raft_consensus.cc:3037] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:18.524991 13391 raft_consensus.cc:3037] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:18.527326 13391 raft_consensus.cc:2454] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f979ca9b58e1474e802d8111ecca17d2 in term 1.
I20260927 18:35:18.527724 13602 leader_election.cc:302] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c5dca3afde407b8ca10a8ad1a136b8, f979ca9b58e1474e802d8111ecca17d2; no voters:
I20260927 18:35:18.527979 13749 raft_consensus.cc:2781] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:18.528076 13536 raft_consensus.cc:2454] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f979ca9b58e1474e802d8111ecca17d2 in term 1.
I20260927 18:35:18.528524 13749 raft_consensus.cc:686] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 1 LEADER]: Becoming Leader. State: Replica: f979ca9b58e1474e802d8111ecca17d2, State: Running, Role: LEADER
I20260927 18:35:18.529069 13749 consensus_queue.cc:237] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } }
I20260927 18:35:18.533439 13238 catalog_manager.cc:5495] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f979ca9b58e1474e802d8111ecca17d2 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "f979ca9b58e1474e802d8111ecca17d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f979ca9b58e1474e802d8111ecca17d2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 35055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 } health_report { overall_health: UNKNOWN } } }
I20260927 18:35:18.614251 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.608663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.0.97.67:35055)
0927 18:35:18.608880 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.608888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.608965 (+ 77us) server_negotiation.cc:407] Connection header received
0927 18:35:18.609242 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.609246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.609337 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.609495 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:18.609913 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.610680 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.612085 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.612479 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.612612 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.613129 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:18.613628 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:18.613793 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.613891 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.613969 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:35:18.622627 13391 raft_consensus.cc:1261] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 1 FOLLOWER]: Refusing update from remote peer f979ca9b58e1474e802d8111ecca17d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:18.622638 13536 raft_consensus.cc:1261] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 1 FOLLOWER]: Refusing update from remote peer f979ca9b58e1474e802d8111ecca17d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:18.623565 13753 consensus_queue.cc:1035] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c5dca3afde407b8ca10a8ad1a136b8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:18.623888 13749 consensus_queue.cc:1035] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c8a3109c8a4308a85388a30132ceaa" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:18.634248 13756 mvcc.cc:205] Tried to move back new op lower bound from 7334027749867839488 to 7334027749499457536. Current Snapshot: MvccSnapshot[applied={T|T < 7334027749867839488}]
I20260927 18:35:18.649384 13238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:18.649658 13238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:18.653347 13238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 816992ed1418436ea02d9406dac612e8: Sending DeleteTablet for 3 replicas of tablet 82c559b16f714ef397f7f2316ff070ac
I20260927 18:35:18.654467 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c559b16f714ef397f7f2316ff070ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:18 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52734
I20260927 18:35:18.654927 13516 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c559b16f714ef397f7f2316ff070ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:18 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47130
I20260927 18:35:18.655524 13371 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c559b16f714ef397f7f2316ff070ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:18 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:43184
I20260927 18:35:18.663373 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13276
I20260927 18:35:18.667512 13768 tablet_replica.cc:330] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: stopping tablet replica
I20260927 18:35:18.669963 13767 tablet_replica.cc:330] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: stopping tablet replica
I20260927 18:35:18.670590 13768 raft_consensus.cc:2229] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:18.671452 13766 tablet_replica.cc:330] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: stopping tablet replica
I20260927 18:35:18.671581 13767 raft_consensus.cc:2229] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:18.671932 13766 raft_consensus.cc:2229] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:35:18.672166 13767 raft_consensus.cc:2258] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:18.672541 13768 raft_consensus.cc:2258] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:18.674458 13767 ts_tablet_manager.cc:1905] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:18.674681 13766 raft_consensus.cc:2258] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:18.675649 13768 ts_tablet_manager.cc:1905] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:18.679380 13766 ts_tablet_manager.cc:1905] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:18.681605 13767 ts_tablet_manager.cc:1918] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:18.681900 13767 log.cc:1196] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/82c559b16f714ef397f7f2316ff070ac
I20260927 18:35:18.682536 13767 ts_tablet_manager.cc:1939] T 82c559b16f714ef397f7f2316ff070ac P 28c8a3109c8a4308a85388a30132ceaa: Deleting consensus metadata
I20260927 18:35:18.682638 13768 ts_tablet_manager.cc:1918] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:18.682878 13768 log.cc:1196] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/82c559b16f714ef397f7f2316ff070ac
I20260927 18:35:18.684697 13226 catalog_manager.cc:4841] TS 28c8a3109c8a4308a85388a30132ceaa (127.0.97.66:41119): tablet 82c559b16f714ef397f7f2316ff070ac (table test-table [id=535b811c46e54b4487ca533dbd8a7f90]) successfully deleted
I20260927 18:35:18.685415 13768 ts_tablet_manager.cc:1939] T 82c559b16f714ef397f7f2316ff070ac P 07c5dca3afde407b8ca10a8ad1a136b8: Deleting consensus metadata
I20260927 18:35:18.688659 13766 ts_tablet_manager.cc:1918] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:18.688915 13766 log.cc:1196] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/82c559b16f714ef397f7f2316ff070ac
I20260927 18:35:18.689495 13766 ts_tablet_manager.cc:1939] T 82c559b16f714ef397f7f2316ff070ac P f979ca9b58e1474e802d8111ecca17d2: Deleting consensus metadata
I20260927 18:35:18.690783 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.664384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.97.65:41101)
0927 18:35:18.664702 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.664711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.666798 (+ 2087us) server_negotiation.cc:407] Connection header received
0927 18:35:18.667783 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.667793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.667924 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.668016 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.670927 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.672121 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.674264 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.674875 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.674992 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.683286 (+ 8294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.683299 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.683319 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.683364 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.687307 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.688283 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.688292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.688295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.688464 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.688949 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.688955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.688959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.689475 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.689841 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.690170 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.690385 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 18:35:18.691495 13224 catalog_manager.cc:4841] TS f979ca9b58e1474e802d8111ecca17d2 (127.0.97.67:35055): tablet 82c559b16f714ef397f7f2316ff070ac (table test-table [id=535b811c46e54b4487ca533dbd8a7f90]) successfully deleted
I20260927 18:35:18.691609 13431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:18.691850 13223 catalog_manager.cc:4841] TS 07c5dca3afde407b8ca10a8ad1a136b8 (127.0.97.65:41101): tablet 82c559b16f714ef397f7f2316ff070ac (table test-table [id=535b811c46e54b4487ca533dbd8a7f90]) successfully deleted
I20260927 18:35:18.798699 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13276
I20260927 18:35:18.815853 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13440
I20260927 18:35:18.830973 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.816881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.0.97.66:41119)
0927 18:35:18.817087 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.817094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.817114 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:35:18.817184 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.817190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.817272 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.817376 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.818690 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.819749 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.821017 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.821681 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.821830 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.824202 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.824219 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.824239 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.824284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.827963 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.828616 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.828626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.828629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.828797 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.829133 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.829138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.829142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.829574 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.829958 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.830375 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.830585 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:35:18.831687 13576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:18.920984 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13440
I20260927 18:35:18.938032 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13585
I20260927 18:35:18.951982 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:18.938842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.97.67:35055)
0927 18:35:18.939057 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:18.939063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:18.939082 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:35:18.939225 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:18.939234 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:18.939310 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:18.939438 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:18.940643 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.941438 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.942784 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:18.943238 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:18.943386 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:18.946004 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:18.946015 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:18.946030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:18.946061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:18.949099 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.949828 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.949839 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.949842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.950046 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:18.950447 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:18.950452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:18.950455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:18.950812 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:18.951154 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:18.951388 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:35:18.951594 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:35:18.952963 13710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:19.087163 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13585
I20260927 18:35:19.108124 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13206
I20260927 18:35:19.109297 13269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:19.221521 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13206
I20260927 18:35:19.245386 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:35:19.580297 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:35:19.580516 389 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:35:19.925611 389 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:35:19.925 UTC [12259] LOG: received smart shutdown request
2026-09-27 18:35:19.929 UTC [12259] LOG: background worker "logical replication launcher" (PID 12268) exited with exit code 1
2026-09-27 18:35:19.929 UTC [12263] LOG: shutting down
2026-09-27 18:35:20.106 UTC [12259] LOG: database system is shut down
2026-09-27T18:35:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53203 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:20 dist-test-slave-n3p4 krb5kdc[13782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:20 dist-test-slave-n3p4 krb5kdc[13782](info): set up 2 sockets
Sep 27 18:35:20 dist-test-slave-n3p4 krb5kdc[13782](info): commencing operation
krb5kdc: starting...
W20260927 18:35:21.346067 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:35:21 dist-test-slave-n3p4 krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:35:21.563673 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39009
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:21.896087 13795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:21.896368 13795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:21.896524 13795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:21.903750 13795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:21.903852 13795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:21.903951 13795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:21.904039 13795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:21.904080 13795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:21.914004 13795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:21.915355 13795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:21.916674 13795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:21.929203 13800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:21.929231 13803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:21.929160 13801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:21.930078 13795 server_base.cc:1034] running on GCE node
I20260927 18:35:21.930809 13795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:21.932219 13795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:21.933444 13795 hybrid_clock.cc:648] HybridClock initialized: now 1790534121933401 us; error 57 us; skew 500 ppm
I20260927 18:35:21.933856 13795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:21 dist-test-slave-n3p4 krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:21.936932 13795 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:21.938364 13795 webserver.cc:460] Webserver started at http://127.0.97.126:39725/ using document root <none> and password file <none>
I20260927 18:35:21.938973 13795 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:21.939069 13795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:21.939304 13795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:21.942111 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "5d98198560cc4b23a923c0677d9c0221"
format_stamp: "Formatted at 2026-09-27 18:35:21 on dist-test-slave-n3p4"
server_key: "720b1ff5662808fb1ff2b146da14c0f2"
server_key_iv: "ef4d4e37ac1c3065435ff8f05bc3cacc"
server_key_version: "encryptionkey@0"
I20260927 18:35:21.942819 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "5d98198560cc4b23a923c0677d9c0221"
format_stamp: "Formatted at 2026-09-27 18:35:21 on dist-test-slave-n3p4"
server_key: "720b1ff5662808fb1ff2b146da14c0f2"
server_key_iv: "ef4d4e37ac1c3065435ff8f05bc3cacc"
server_key_version: "encryptionkey@0"
I20260927 18:35:21.947620 13795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:35:21.950784 13810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:21.952183 13795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:21.952328 13795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5d98198560cc4b23a923c0677d9c0221"
format_stamp: "Formatted at 2026-09-27 18:35:21 on dist-test-slave-n3p4"
server_key: "720b1ff5662808fb1ff2b146da14c0f2"
server_key_iv: "ef4d4e37ac1c3065435ff8f05bc3cacc"
server_key_version: "encryptionkey@0"
I20260927 18:35:21.952493 13795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:21.975613 13795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:21.979835 13795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:21.980060 13795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:21.999001 13795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:39009
I20260927 18:35:21.999091 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:39009 every 8 connection(s)
I20260927 18:35:22.000329 13795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:22.004029 13862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:22.004359 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13795
I20260927 18:35:22.004572 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:35:22.005234 389 external_mini_cluster.cc:1428] Setting key 582135df4c0222d135d89b6cf03eead8
I20260927 18:35:22.010499 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:22.035113 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.013632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39009 (local address 127.0.0.1:60588)
0927 18:35:22.014576 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.014619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:22.016529 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:22.017302 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:22.017310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:22.018048 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:22.018366 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.018383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.019887 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.019892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.020935 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.020946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.021864 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.021871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.022099 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.022799 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:22.022820 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:22.027651 (+ 4831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:22.030999 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.031015 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.031031 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.031612 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.032039 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.032047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.032053 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.032267 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.033104 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:22.033114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:22.033381 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 18:35:22.033672 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 18:35:22.034107 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":2,"thread_start_us":475,"threads_started":1}
I20260927 18:35:22.035161 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.013635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.0.97.126:39009)
0927 18:35:22.015365 (+ 1730us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:22.015377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:22.015424 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:35:22.016881 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:22.016886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:22.017089 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:22.017286 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:22.018562 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.019736 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.021072 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.021602 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.021830 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.027897 (+ 6067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:22.027905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:22.027916 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:22.027944 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:22.030863 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.031768 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.031774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.031776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.031925 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.032423 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.032427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.032429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.032952 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:22.033132 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:22.033890 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 18:35:22.034284 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":1519,"spinlock_wait_cycles":22272,"thread_start_us":1310,"threads_started":1}
I20260927 18:35:22.038410 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:35:22.039565 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.039182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51770)
0927 18:35:22.039344 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.039447 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:35:22.039829 13814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
I20260927 18:35:22.041291 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.040976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51776)
0927 18:35:22.041119 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.041182 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:35:22.041476 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44953: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260927 18:35:22.053095 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:44953
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:22.090797 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } attempt: 1
I20260927 18:35:22.092006 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.091552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51782)
0927 18:35:22.091761 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.091864 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:35:22.092993 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.092656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51792)
0927 18:35:22.092794 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.092873 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:35:22.093171 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44953: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
I20260927 18:35:22.161423 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } attempt: 2
I20260927 18:35:22.162772 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.162216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51794)
0927 18:35:22.162510 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.162621 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:35:22.163894 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.163494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51804)
0927 18:35:22.163651 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.163743 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:35:22.164108 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44953: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
I20260927 18:35:22.255379 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } attempt: 3
I20260927 18:35:22.256569 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.256107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51808)
0927 18:35:22.256328 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.256436 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:35:22.257616 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.257281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51820)
0927 18:35:22.257415 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.257484 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:35:22.257826 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44953: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
W20260927 18:35:22.378031 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:22.378369 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:22.378547 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:22.385890 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:22.386014 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:22.386118 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:22.386217 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:22.386283 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:22.396047 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:22.397333 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:22.398682 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:35:22.401079 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } attempt: 4
I20260927 18:35:22.402375 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.401760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51826)
0927 18:35:22.402132 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.402242 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260927 18:35:22.403496 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.403047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51832)
0927 18:35:22.403203 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.403321 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 18:35:22.403765 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44953: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44953: connect: Connection refused (error 111)
W20260927 18:35:22.410629 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:22.410684 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:22.410758 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:22.411386 13869 server_base.cc:1034] running on GCE node
I20260927 18:35:22.412091 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:22.413424 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:22.414615 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790534122414558 us; error 51 us; skew 500 ppm
I20260927 18:35:22.414937 13869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:22.418188 13869 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260927 18:35:22.419620 13869 webserver.cc:460] Webserver started at http://127.0.97.125:44441/ using document root <none> and password file <none>
I20260927 18:35:22.420308 13869 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:22.420428 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:22.420771 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:22.423893 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data/instance:
uuid: "edaceb42a1594aa1a1bf8c3a2026526f"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "2fe435fbd02fedb836378a41244d9c9e"
server_key_iv: "0ae7e7905e78f60caf05eeec18617f65"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.424810 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal/instance:
uuid: "edaceb42a1594aa1a1bf8c3a2026526f"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "2fe435fbd02fedb836378a41244d9c9e"
server_key_iv: "0ae7e7905e78f60caf05eeec18617f65"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.430106 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:35:22.433593 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:22.435063 13869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:35:22.435256 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "edaceb42a1594aa1a1bf8c3a2026526f"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "2fe435fbd02fedb836378a41244d9c9e"
server_key_iv: "0ae7e7905e78f60caf05eeec18617f65"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.435452 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:22.460983 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:22.465373 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:22.465654 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:22.484956 13869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:44953
I20260927 18:35:22.484987 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:44953 every 8 connection(s)
I20260927 18:35:22.486382 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
I20260927 18:35:22.490144 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:22.494953 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13869
I20260927 18:35:22.495167 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-1/wal/instance
I20260927 18:35:22.495684 389 external_mini_cluster.cc:1428] Setting key 05ce1fd1fa05c7921c1da06b0e67b6b4
I20260927 18:35:22.497164 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
I20260927 18:35:22.521723 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.500537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.97.126:39009)
0927 18:35:22.500794 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:22.500803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:22.502113 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 18:35:22.504078 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:22.504083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:22.504147 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:22.504247 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:22.506377 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.507057 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.509583 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.510088 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.510189 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.514578 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:22.514588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:22.514595 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:22.514650 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:22.518522 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.519321 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.519325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.519328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.519454 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.519913 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.519918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.519920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.520219 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:22.520376 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:22.521171 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 18:35:22.521397 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:35:22.522708 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.501010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39009 (local address 127.0.0.1:60596)
0927 18:35:22.501919 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.501967 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:22.503766 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:22.504521 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:22.504536 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:22.505343 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:22.506187 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.506214 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.507232 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.507241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.509434 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.509453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.510332 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.510343 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.510676 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.511456 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:22.511531 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:22.514340 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:22.518680 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.518687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.518700 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.519175 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.519556 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.519559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.519561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.519775 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.520347 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:22.520355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:22.520713 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 18:35:22.521045 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 18:35:22.521602 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":416,"threads_started":1}
I20260927 18:35:22.524616 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:35:22.540930 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.525672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.0.97.125:44953)
0927 18:35:22.526459 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:22.526470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:22.526504 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:35:22.526564 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:22.526584 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:22.526836 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:22.526997 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:22.527654 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.529389 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.530591 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.531146 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.531285 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.534838 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:22.534849 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:22.534863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:22.534892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:22.537634 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.538482 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.538489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.538491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.538723 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.539199 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.539202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.539204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.539591 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:22.539846 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:22.540340 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:35:22.540626 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":588,"thread_start_us":438,"threads_started":1}
I20260927 18:35:22.541024 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.525420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51840)
0927 18:35:22.525643 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.525662 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:22.525816 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:22.526999 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:22.527002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:22.527363 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:22.527494 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.527501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.529521 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.529528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.530446 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.530458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.531292 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.531297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.531496 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.532141 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:22.532157 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:22.534609 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:22.537769 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.537821 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.537827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.538361 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.538866 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.538872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.538877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.539076 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.539758 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:22.539765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:22.539970 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 18:35:22.540139 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:35:22.540488 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:35:22.543835 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:35:22.545091 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.544570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55628)
0927 18:35:22.544757 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.544880 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 18:35:22.545428 13886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.547277 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.546809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55636)
0927 18:35:22.547069 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.547154 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 18:35:22.547648 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260927 18:35:22.552759 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:41663
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:22.597019 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 1
I20260927 18:35:22.598433 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.597960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55648)
0927 18:35:22.598161 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.598266 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 18:35:22.599431 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.599100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55652)
0927 18:35:22.599240 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.599324 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:35:22.599617 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.667869 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 2
I20260927 18:35:22.669307 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.668794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55662)
0927 18:35:22.669017 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.669142 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:35:22.670446 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.670083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55668)
0927 18:35:22.670236 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.670321 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:35:22.670658 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.703104 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } attempt: 5
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:35:22.717211 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.703926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51850)
0927 18:35:22.704158 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.704181 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:22.704327 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:22.704708 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:22.704714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:22.705155 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:22.705331 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.705340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.706456 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.706461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.707357 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:22.707368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.708113 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.708119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:22.708295 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.708982 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:22.709000 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:22.711397 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:22.715019 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.715022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.715025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.715395 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.715777 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:22.715781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:22.715783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:22.715908 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:22.716496 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:22.716499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:22.716653 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 18:35:22.716776 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:35:22.716964 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 18:35:22.717466 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.704002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.125:44953)
0927 18:35:22.704263 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:22.704270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:22.704291 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:35:22.704482 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:22.704493 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:22.704562 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:22.704670 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:22.705497 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.706311 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.707495 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:22.707956 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:22.708099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:22.711596 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:22.711602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:22.711607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:22.711658 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:22.714882 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.715510 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.715516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.715519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.715671 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:22.716006 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:22.716011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:22.716014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:22.716367 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:22.716517 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:22.716914 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 18:35:22.717130 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 18:35:22.718540 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:35:22.719920 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.719416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55672)
0927 18:35:22.719613 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.719734 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:35:22.721014 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.720594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55688)
0927 18:35:22.720752 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.720864 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 18:35:22.721287 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.761874 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 3
I20260927 18:35:22.763137 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.762542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55698)
0927 18:35:22.762821 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.762935 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:35:22.764362 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.763797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55710)
0927 18:35:22.764081 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.764191 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260927 18:35:22.764670 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.772548 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 1
I20260927 18:35:22.773677 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.773145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55714)
0927 18:35:22.773347 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.773465 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 18:35:22.774756 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.774342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55726)
0927 18:35:22.774504 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.774596 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 18:35:22.774956 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.843189 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 2
I20260927 18:35:22.844507 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.843894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55740)
0927 18:35:22.844157 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.844290 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 18:35:22.845621 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.845211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55752)
0927 18:35:22.845380 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.845473 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:35:22.845904 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
W20260927 18:35:22.879993 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:22.880257 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:22.880437 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:22.887415 13943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:22.887514 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:22.887598 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:22.887687 13943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:22.887775 13943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:22.897287 13943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39009,127.0.97.125:44953,127.0.97.124:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:22.898612 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:22.899899 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:35:22.907895 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 4
I20260927 18:35:22.909020 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.908543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55758)
0927 18:35:22.908752 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.908839 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 18:35:22.910107 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.909707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55762)
0927 18:35:22.909869 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.909946 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:35:22.910451 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
W20260927 18:35:22.911862 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:22.911922 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:22.911899 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:22.913359 13943 server_base.cc:1034] running on GCE node
I20260927 18:35:22.914206 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:22.915613 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:22.916877 13943 hybrid_clock.cc:648] HybridClock initialized: now 1790534122916826 us; error 41 us; skew 500 ppm
I20260927 18:35:22.917286 13943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:22 dist-test-slave-n3p4 krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:22.920454 13943 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260927 18:35:22.921972 13943 webserver.cc:460] Webserver started at http://127.0.97.124:43017/ using document root <none> and password file <none>
I20260927 18:35:22.922664 13943 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:22.922807 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:22.923110 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:22.926139 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data/instance:
uuid: "6cae41d661e342b3a970fc2b8882ab23"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "9e49d395a2075fe3ee472bfb264a11e7"
server_key_iv: "d44c558a714a026685bdf4bb3c02860d"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.926923 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal/instance:
uuid: "6cae41d661e342b3a970fc2b8882ab23"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "9e49d395a2075fe3ee472bfb264a11e7"
server_key_iv: "d44c558a714a026685bdf4bb3c02860d"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.932293 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:35:22.935681 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:22.936964 13943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:22.937168 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "6cae41d661e342b3a970fc2b8882ab23"
format_stamp: "Formatted at 2026-09-27 18:35:22 on dist-test-slave-n3p4"
server_key: "9e49d395a2075fe3ee472bfb264a11e7"
server_key_iv: "d44c558a714a026685bdf4bb3c02860d"
server_key_version: "encryptionkey@0"
I20260927 18:35:22.937379 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:22.952153 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 3
I20260927 18:35:22.953200 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.952739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55774)
0927 18:35:22.952932 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.953050 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:35:22.954316 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.953897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55790)
0927 18:35:22.954136 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:22.954224 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 18:35:22.954519 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:41663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:41663: connect: Connection refused (error 111)
I20260927 18:35:22.965749 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:22.970371 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:22.970680 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:22.989396 13943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:41663
I20260927 18:35:22.989429 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:41663 every 8 connection(s)
I20260927 18:35:22.990748 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
I20260927 18:35:22.994421 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:22.994849 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 13943
I20260927 18:35:22.995039 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790533919012461-389-0/minicluster-data/master-2/wal/instance
I20260927 18:35:22.995607 389 external_mini_cluster.cc:1428] Setting key b463f9bf882d75c9c46d01d10c603bcd
I20260927 18:35:23.004716 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:23.020843 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:22.997580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.0.97.124:41663)
0927 18:35:22.998645 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:22.998658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:22.998702 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:35:23.001302 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.001323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.001693 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.002192 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.003670 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.004921 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.006644 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.007324 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.007525 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.012652 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.012661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.012674 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.012721 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.016229 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.017055 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.017062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.017080 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.017267 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.017643 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.017649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.017651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.018411 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.018711 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.019095 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.019494 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":643,"threads_started":1}
I20260927 18:35:23.023398 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.007977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.97.126:39009)
0927 18:35:23.008222 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.008231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.009002 (+ 771us) server_negotiation.cc:407] Connection header received
0927 18:35:23.009258 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.009263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.009341 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.009447 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.010581 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.011277 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.012632 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.013046 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.013136 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.017052 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.017058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.017063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.017101 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.020580 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.021341 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.021348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.021354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.021496 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.021928 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.021934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.021938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.022234 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.022388 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.022824 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.023039 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:35:23.023416 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.007978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39009 (local address 127.0.0.1:60604)
0927 18:35:23.008866 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:23.008906 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:23.009054 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:23.009504 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:23.009511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:23.010176 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:23.010426 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.010435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.011422 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.011427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.012487 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.012502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.013249 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.013258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.013413 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.014036 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:23.014060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:23.016843 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:23.020700 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.020708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.020713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.021223 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.021610 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.021616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.021620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.021762 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.022363 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:23.022370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:23.022550 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 18:35:23.022722 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 18:35:23.022992 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":508,"threads_started":1}
I20260927 18:35:23.024783 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:35:23.037582 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.023717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.0.97.126:39009)
0927 18:35:23.023856 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.023862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.024147 (+ 285us) server_negotiation.cc:407] Connection header received
0927 18:35:23.024326 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.024331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.024398 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.024492 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.025768 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.026496 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.027720 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.028126 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.028261 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.032505 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.032510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.032514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.032547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.035061 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.035705 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.035712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.035719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.035839 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.036193 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.036198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.036202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.036457 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.036742 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.037005 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.037229 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:35:23.037914 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.025371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44953 (local address 127.0.0.1:51862)
0927 18:35:23.025679 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:23.025698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:23.025834 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:23.026160 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:23.026163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:23.026500 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:23.026634 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.026641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.027617 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.027622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.028464 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.028474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.029426 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.029431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.029661 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.030239 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:23.030254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:23.032454 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:23.035642 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.035649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.035652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.036001 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.036373 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.036379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.036382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.036537 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.036997 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:23.037002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:23.037148 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 18:35:23.037284 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:35:23.037545 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 18:35:23.037964 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.025513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.97.125:44953)
0927 18:35:23.025673 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.025679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.025744 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:35:23.025973 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.025977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.026043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.026142 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.026784 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.027476 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.028599 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.029275 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.029415 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.032645 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.032649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.032652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.032683 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.035513 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.036116 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.036122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.036125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.036267 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.036643 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.036649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.036653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.036884 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.037054 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.037364 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.037553 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:35:23.038947 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:35:23.052026 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.039701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.0.97.124:41663)
0927 18:35:23.039849 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.039854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.040040 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:35:23.040191 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.040195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.040249 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.040342 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.041001 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.041750 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.042947 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.043351 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.043484 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.046385 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.046389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.046393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.046418 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.049527 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.050277 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.050285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.050288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.050448 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.050799 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.050807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.050812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.051150 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.051298 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.051624 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.051782 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:35:23.052428 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.039737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55810)
0927 18:35:23.039974 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:23.039991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:23.040093 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:23.040355 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:23.040359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:23.040707 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:23.040867 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.040878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.041916 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.041924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.042817 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.042828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.043497 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.043503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.043693 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.044226 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:23.044243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:23.046225 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:23.049703 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.049709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.049714 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.050162 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.050561 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.050571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.050574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.050691 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.051264 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:23.051267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:23.051381 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 18:35:23.051531 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:35:23.051986 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":90}
I20260927 18:35:23.054322 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.039237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.0.97.125:44953)
0927 18:35:23.039416 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.039421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.039463 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:35:23.039621 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.039626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.039695 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.039825 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.041232 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.042079 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.043337 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.043741 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.043870 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.048051 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.048057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.048061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.048099 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.051399 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.052277 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.052282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.052286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.052445 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.052875 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.052882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.052884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.053139 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.053349 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.053734 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.053987 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:35:23.058763 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: Bootstrap starting.
I20260927 18:35:23.059267 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:35:23.062090 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:23.063129 14010 log.cc:824] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:23.066699 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: No bootstrap required, opened a new log
I20260927 18:35:23.080057 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.080840 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:23.081024 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cae41d661e342b3a970fc2b8882ab23, State: Initialized, Role: FOLLOWER
I20260927 18:35:23.081966 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.085004 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.064915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.0.97.126:39009)
0927 18:35:23.065191 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.065196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.065689 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:35:23.065992 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.065998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.066075 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.066148 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.068773 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.070017 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.072426 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.073074 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.073168 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.077297 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.077304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.077309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.077358 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.081159 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.082299 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.082306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.082312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.082487 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.083081 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.083087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.083091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.083483 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.083730 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.084310 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.084577 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 18:35:23.086041 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.064182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.97.125:44953)
0927 18:35:23.064357 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.064365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.065744 (+ 1379us) server_negotiation.cc:407] Connection header received
0927 18:35:23.066006 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.066010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.066089 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.066152 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.068773 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.069914 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.072418 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.073081 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.073214 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.077451 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.077473 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.077478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.077520 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.081233 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.082335 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.082340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.082343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.082495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.083149 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.083158 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.083164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.083639 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.083935 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.085136 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.085472 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:35:23.087916 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:23.088959 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.089632 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.091773 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.064995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.0.97.124:41663)
0927 18:35:23.065313 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.065317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.072723 (+ 7406us) server_negotiation.cc:407] Connection header received
0927 18:35:23.072963 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.072970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.073064 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.073139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.075759 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.077023 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.079134 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.079913 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.080009 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.083826 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.083833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.083837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.083881 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.088334 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.089235 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.089245 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.089250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.089492 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.089976 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.089982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.089985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.090438 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.090756 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.091044 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.091282 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
W20260927 18:35:23.099452 14034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:35:23.099612 14034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:35:23.095166 13943 master_runner.cc:186] Error getting master registration for 127.0.97.126:39009: OK, instance_id { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" instance_seqno: 1790534121993617 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:35:23.103466 13943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:35:23.121625 14023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:23.121984 14023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.123639 14023 leader_election.cc:288] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d98198560cc4b23a923c0677d9c0221 (127.0.97.126:39009), edaceb42a1594aa1a1bf8c3a2026526f (127.0.97.125:44953)
I20260927 18:35:23.131740 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 4
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260927 18:35:23.147665 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.132255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55826)
0927 18:35:23.132495 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:23.132515 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:23.132628 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:23.133058 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:23.133065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:23.133578 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:23.133814 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.133826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.134839 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.134846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.136299 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.136313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.137084 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.137091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.137289 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.138019 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:23.138042 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:23.140957 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:23.145009 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.145014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.145018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.145368 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.145808 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.145813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.145817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.145999 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.146654 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:23.146661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:23.146835 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 18:35:23.146974 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:35:23.147268 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 18:35:23.148053 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.132460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.0.97.124:41663)
0927 18:35:23.132693 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.132700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.132722 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:35:23.132815 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.132819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.132907 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.133021 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.133953 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.134697 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.136462 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.136929 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.137084 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.141124 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.141130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.141137 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.141173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.144893 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.145483 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.145490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.145494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.145656 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.146154 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.146160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.146163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.146523 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.146677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.147182 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.147587 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:35:23.154733 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: Bootstrap starting.
I20260927 18:35:23.158828 13862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:23.160118 13862 log.cc:824] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:23.163472 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: No bootstrap required, opened a new log
I20260927 18:35:23.176910 13862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.177338 13862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:23.177464 13862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d98198560cc4b23a923c0677d9c0221, State: Initialized, Role: FOLLOWER
I20260927 18:35:23.181134 13862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.182597 14037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.183002 14037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.183743 13862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:23.183979 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cae41d661e342b3a970fc2b8882ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d98198560cc4b23a923c0677d9c0221" is_pre_election: true
I20260927 18:35:23.184571 13837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cae41d661e342b3a970fc2b8882ab23 in term 0.
I20260927 18:35:23.185230 13961 leader_election.cc:302] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d98198560cc4b23a923c0677d9c0221, 6cae41d661e342b3a970fc2b8882ab23; no voters:
I20260927 18:35:23.185680 14023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:23.185870 14023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:23.186013 14023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:23.189433 14023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.191452 14023 leader_election.cc:288] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [CANDIDATE]: Term 1 election: Requested vote from peers 5d98198560cc4b23a923c0677d9c0221 (127.0.97.126:39009), edaceb42a1594aa1a1bf8c3a2026526f (127.0.97.125:44953)
I20260927 18:35:23.192597 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cae41d661e342b3a970fc2b8882ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d98198560cc4b23a923c0677d9c0221"
I20260927 18:35:23.192837 13837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:23.196157 13837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cae41d661e342b3a970fc2b8882ab23 in term 1.
I20260927 18:35:23.196676 13961 leader_election.cc:302] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d98198560cc4b23a923c0677d9c0221, 6cae41d661e342b3a970fc2b8882ab23; no voters:
I20260927 18:35:23.197060 14023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:23.199152 14023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [term 1 LEADER]: Becoming Leader. State: Replica: 6cae41d661e342b3a970fc2b8882ab23, State: Running, Role: LEADER
I20260927 18:35:23.199831 14023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.209645 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } attempt: 5
I20260927 18:35:23.216638 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cae41d661e342b3a970fc2b8882ab23. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.219806 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: This master's current role is: LEADER
Sep 27 18:35:23 dist-test-slave-n3p4 krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
W20260927 18:35:23.222692 13964 tablet.cc:2369] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:23.222769 14050 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:35:23.231427 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.210219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:41663 (local address 127.0.0.1:55834)
0927 18:35:23.210539 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:23.210558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:23.210693 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:23.211353 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:23.211361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:23.211907 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:23.212114 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.212127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.214358 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.214363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.215749 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:23.215764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.216904 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.216910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:23.217124 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.217928 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:23.217953 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:23.221014 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:23.227670 (+ 6656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.227675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.227680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.228221 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.229000 (+ 779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:23.229005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:23.229009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:23.229207 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:23.230458 (+ 1251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:23.230463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:23.230635 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 18:35:23.230762 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 18:35:23.231036 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 18:35:23.232679 14051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 18:35:23.232995 13816 tablet.cc:2369] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:23.233366 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:23.210889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.0.97.124:41663)
0927 18:35:23.211050 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:23.211056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:23.211083 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:35:23.211133 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:23.211139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:23.211213 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:23.211307 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:23.212656 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.213936 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.215914 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:23.216730 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:23.216901 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:23.221267 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:23.221276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:23.221284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:23.221336 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:23.227460 (+ 6124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.228477 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.228484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.228489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.228813 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:23.229409 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:23.229447 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:23.229450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:23.230234 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:23.231759 (+ 1525us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:23.232005 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:35:23.232496 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:35:23.227193 13795 master_runner.cc:186] Error getting master registration for 127.0.97.125:44953: OK, instance_id { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" instance_seqno: 1790534122479517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:35:23.237541 13795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:35:23.239238 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: Bootstrap starting.
I20260927 18:35:23.243184 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:23.243544 14051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:23.244499 13936 log.cc:824] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:23.251443 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: No bootstrap required, opened a new log
I20260927 18:35:23.270917 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.271891 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:23.272006 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaceb42a1594aa1a1bf8c3a2026526f, State: Initialized, Role: FOLLOWER
I20260927 18:35:23.272646 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } }
I20260927 18:35:23.273000 13837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [term 1 FOLLOWER]: Refusing update from remote peer 6cae41d661e342b3a970fc2b8882ab23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:23.274096 14045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:23.273890 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.274904 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cae41d661e342b3a970fc2b8882ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaceb42a1594aa1a1bf8c3a2026526f"
I20260927 18:35:23.274937 13911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cae41d661e342b3a970fc2b8882ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" is_pre_election: true
I20260927 18:35:23.275274 13909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:23.274672 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.275383 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:23.284601 13909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [term 1 FOLLOWER]: Refusing update from remote peer 6cae41d661e342b3a970fc2b8882ab23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:23.285734 14045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:23.298137 14037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cae41d661e342b3a970fc2b8882ab23. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.298548 14037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.301482 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cae41d661e342b3a970fc2b8882ab23. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.301925 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.302625 14053 mvcc.cc:205] Tried to move back new op lower bound from 7334027768913727488 to 7334027768634126336. Current Snapshot: MvccSnapshot[applied={T|T < 7334027768913727488}]
I20260927 18:35:23.304118 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.304391 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:23.305317 14051 catalog_manager.cc:1347] Generated new cluster ID: 71686a5107a74e1ba415fbea7ceed549
I20260927 18:35:23.305435 14051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:23.315321 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.315562 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.317924 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.318176 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:23.320461 14037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cae41d661e342b3a970fc2b8882ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d98198560cc4b23a923c0677d9c0221" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39009 } } peers { permanent_uuid: "edaceb42a1594aa1a1bf8c3a2026526f" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44953 } } peers { permanent_uuid: "6cae41d661e342b3a970fc2b8882ab23" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 41663 } } }
I20260927 18:35:23.320731 14037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:35:23.335152 14051 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:23.337630 14051 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:23.358542 14051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cae41d661e342b3a970fc2b8882ab23: Generated new TSK 0
I20260927 18:35:23.359500 14051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:35:23.371577 14073 catalog_manager.cc:1259] Loaded cluster ID: 71686a5107a74e1ba415fbea7ceed549
I20260927 18:35:23.371685 14073 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: loading cluster ID for follower catalog manager: success
I20260927 18:35:23.377882 14073 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: acquiring CA information for follower catalog manager: success
I20260927 18:35:23.381213 14073 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edaceb42a1594aa1a1bf8c3a2026526f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:35:24.227048 14050 catalog_manager.cc:1259] Loaded cluster ID: 71686a5107a74e1ba415fbea7ceed549
I20260927 18:35:24.227202 14050 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: loading cluster ID for follower catalog manager: success
I20260927 18:35:24.229719 14050 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: acquiring CA information for follower catalog manager: success
I20260927 18:35:24.231542 14050 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d98198560cc4b23a923c0677d9c0221: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:35:24.451113 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13795
I20260927 18:35:24.452513 13857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:24.567502 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13795
I20260927 18:35:24.586817 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13869
I20260927 18:35:24.588092 13931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:24.697096 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13869
I20260927 18:35:24.717095 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 13943
I20260927 18:35:24.718186 14005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:24.827261 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 13943
2026-09-27T18:35:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4581 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:24 dist-test-slave-n3p4 krb5kdc[14088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:24 dist-test-slave-n3p4 krb5kdc[14088](info): set up 2 sockets
Sep 27 18:35:24 dist-test-slave-n3p4 krb5kdc[14088](info): commencing operation
krb5kdc: starting...
W20260927 18:35:25.938423 389 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:35:26.169600 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43783
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:26.498672 14101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:26.498976 14101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:26.499114 14101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:26.506283 14101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:26.506389 14101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:26.506479 14101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:26.506587 14101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:26.506665 14101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:26.516700 14101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:26.518134 14101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:26.519446 14101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:26.531345 14106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:26.531287 14107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:26.531395 14109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:26.535429 14101 server_base.cc:1034] running on GCE node
I20260927 18:35:26.536397 14101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:26.538214 14101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:26.539474 14101 hybrid_clock.cc:648] HybridClock initialized: now 1790534126539429 us; error 58 us; skew 500 ppm
I20260927 18:35:26.539899 14101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:26.543664 14101 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:26.545248 14101 webserver.cc:460] Webserver started at http://127.0.97.126:45405/ using document root <none> and password file <none>
I20260927 18:35:26.546079 14101 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:26.546223 14101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:26.546543 14101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:26.549762 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "3bde7d02224c4bf1af90f33fa1bb3032"
format_stamp: "Formatted at 2026-09-27 18:35:26 on dist-test-slave-n3p4"
server_key: "567a4832018a70e20a6609b43548e2e5"
server_key_iv: "f791722f309c86c3c15123623b316660"
server_key_version: "encryptionkey@0"
I20260927 18:35:26.550863 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "3bde7d02224c4bf1af90f33fa1bb3032"
format_stamp: "Formatted at 2026-09-27 18:35:26 on dist-test-slave-n3p4"
server_key: "567a4832018a70e20a6609b43548e2e5"
server_key_iv: "f791722f309c86c3c15123623b316660"
server_key_version: "encryptionkey@0"
I20260927 18:35:26.556439 14101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:35:26.560214 14116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:26.561642 14101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:26.561888 14101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "3bde7d02224c4bf1af90f33fa1bb3032"
format_stamp: "Formatted at 2026-09-27 18:35:26 on dist-test-slave-n3p4"
server_key: "567a4832018a70e20a6609b43548e2e5"
server_key_iv: "f791722f309c86c3c15123623b316660"
server_key_version: "encryptionkey@0"
I20260927 18:35:26.562104 14101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:26.586759 14101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:26.591161 14101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:26.591526 14101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:26.610929 14101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43783
I20260927 18:35:26.611076 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43783 every 8 connection(s)
I20260927 18:35:26.612332 14101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:26.616381 14168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:26.622202 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14101
I20260927 18:35:26.622421 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:35:26.623049 389 external_mini_cluster.cc:1428] Setting key 7c5062182ba05ac8204c239e1f62c8cf
I20260927 18:35:26.628278 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032: Bootstrap starting.
I20260927 18:35:26.631620 14168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:26.632685 14168 log.cc:824] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:26.635660 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032: No bootstrap required, opened a new log
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:26.645346 14168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } }
I20260927 18:35:26.645848 14168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:26.646010 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:26.624981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.97.126:43783)
0927 18:35:26.625941 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:26.625952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:26.625994 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:35:26.627726 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:26.627749 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:26.628134 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:26.628523 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:26.630013 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:26.631307 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:26.632662 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:26.633393 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:26.633601 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:26.637736 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:26.637744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:26.637758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:26.637848 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:26.641186 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:26.642163 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:26.642181 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:26.642189 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:26.642437 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:26.642915 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:26.642920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:26.642924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:26.643537 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:26.643882 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:26.644364 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 18:35:26.644824 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":473,"threads_started":1}
I20260927 18:35:26.646005 14168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bde7d02224c4bf1af90f33fa1bb3032, State: Initialized, Role: FOLLOWER
I20260927 18:35:26.646754 14168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } }
I20260927 18:35:26.647023 14168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:26.647186 14168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:26.647425 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:26.650379 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:35:26.650928 14168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } }
I20260927 18:35:26.651456 14168 leader_election.cc:302] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bde7d02224c4bf1af90f33fa1bb3032; no voters:
I20260927 18:35:26.652103 14168 leader_election.cc:288] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:26.652297 14173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:26.652947 14173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [term 1 LEADER]: Becoming Leader. State: Replica: 3bde7d02224c4bf1af90f33fa1bb3032, State: Running, Role: LEADER
I20260927 18:35:26.653592 14173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } }
I20260927 18:35:26.655507 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bde7d02224c4bf1af90f33fa1bb3032. Latest consensus state: current_term: 1 leader_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } } }
I20260927 18:35:26.655783 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:26.656512 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bde7d02224c4bf1af90f33fa1bb3032" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43783 } } }
I20260927 18:35:26.656977 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:26.660331 14168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:26.662600 14177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:26.668586 14177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:26.677258 14177 catalog_manager.cc:1347] Generated new cluster ID: 0b92f2eebb6f48a6917e5b4c5a4c0478
I20260927 18:35:26.677369 14177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:26.696179 14177 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:26.697618 14177 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:26.705680 14177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bde7d02224c4bf1af90f33fa1bb3032: Generated new TSK 0
I20260927 18:35:26.706615 14177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:35:26.763614 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14101
I20260927 18:35:26.764905 14163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:26.863023 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14101
2026-09-27T18:35:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2031 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14196](info): set up 2 sockets
Sep 27 18:35:26 dist-test-slave-n3p4 krb5kdc[14196](info): commencing operation
krb5kdc: starting...
W20260927 18:35:27.970270 389 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:35:28 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:35:28.206885 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:28.532516 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:28.532804 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:28.532955 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:28.540158 14209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:28.540258 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:28.540365 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:28.540463 14209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:28.540529 14209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:28.550185 14209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:28.551445 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:28.552760 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:28.564389 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:28.564355 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:28.564427 14217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:28.565096 14209 server_base.cc:1034] running on GCE node
I20260927 18:35:28.565907 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:28.567376 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:28.568608 14209 hybrid_clock.cc:648] HybridClock initialized: now 1790534128568544 us; error 66 us; skew 500 ppm
I20260927 18:35:28.568981 14209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:28 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:28.572170 14209 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:28.573659 14209 webserver.cc:460] Webserver started at http://127.0.97.126:44171/ using document root <none> and password file <none>
I20260927 18:35:28.574405 14209 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:28.574553 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:28.574862 14209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:28.577971 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "fe165813fc0b4cdebc03a4ce94149ad3"
format_stamp: "Formatted at 2026-09-27 18:35:28 on dist-test-slave-n3p4"
server_key: "272b3bae4f2c3cf9c8c64fbfbc345a9e"
server_key_iv: "ff4d985243f6a16219b72aa2f58a19bb"
server_key_version: "encryptionkey@0"
I20260927 18:35:28.578722 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "fe165813fc0b4cdebc03a4ce94149ad3"
format_stamp: "Formatted at 2026-09-27 18:35:28 on dist-test-slave-n3p4"
server_key: "272b3bae4f2c3cf9c8c64fbfbc345a9e"
server_key_iv: "ff4d985243f6a16219b72aa2f58a19bb"
server_key_version: "encryptionkey@0"
I20260927 18:35:28.583779 14209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:35:28.587628 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:28.588991 14209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:28.589236 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "fe165813fc0b4cdebc03a4ce94149ad3"
format_stamp: "Formatted at 2026-09-27 18:35:28 on dist-test-slave-n3p4"
server_key: "272b3bae4f2c3cf9c8c64fbfbc345a9e"
server_key_iv: "ff4d985243f6a16219b72aa2f58a19bb"
server_key_version: "encryptionkey@0"
I20260927 18:35:28.589442 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:28.606261 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:28.610649 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:28.610953 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:28.630401 14209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37019
I20260927 18:35:28.630494 14275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37019 every 8 connection(s)
I20260927 18:35:28.631780 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:28.635560 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:28.638445 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14209
I20260927 18:35:28.638657 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:35:28.639487 389 external_mini_cluster.cc:1428] Setting key 0d011184650616d3e2ec6595961e70b4
I20260927 18:35:28.647473 14276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap starting.
I20260927 18:35:28.650488 14276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:28.651453 14276 log.cc:824] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:35:28 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:28.654485 14276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: No bootstrap required, opened a new log
I20260927 18:35:28.662366 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:28.641696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.0.97.126:37019)
0927 18:35:28.642477 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:28.642489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:28.642528 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:35:28.644182 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:28.644203 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:28.644590 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:28.645016 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:28.646520 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:28.647875 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:28.649244 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:28.649962 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:28.650149 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:28.654720 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:28.654736 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:28.654750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:28.654783 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:28.658108 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:28.658794 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:28.658800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:28.658806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:28.658975 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:28.659386 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:28.659393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:28.659403 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:28.659939 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:28.660378 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:28.660936 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 18:35:28.661352 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":350,"threads_started":1}
I20260927 18:35:28.662451 14276 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:28.662896 14276 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:28.663064 14276 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Initialized, Role: FOLLOWER
I20260927 18:35:28.663595 14276 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:28.663842 14276 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:28.664002 14276 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:28.664198 14276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:28.666625 14276 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:28.667097 14276 leader_election.cc:302] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe165813fc0b4cdebc03a4ce94149ad3; no voters:
I20260927 18:35:28.667662 14276 leader_election.cc:288] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:28.667786 14281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:28.668462 14281 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 LEADER]: Becoming Leader. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Running, Role: LEADER
I20260927 18:35:28.668937 14281 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:28.669701 14276 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:28.671784 14283 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe165813fc0b4cdebc03a4ce94149ad3. Latest consensus state: current_term: 1 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:28.672204 14283 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:28.672091 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:28.672405 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:28.674002 14286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:28.680362 14286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:28.691190 14286 catalog_manager.cc:1347] Generated new cluster ID: b6413f752c024fa799f515e98e9d5737
I20260927 18:35:28.691324 14286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:28.706213 14286 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:28.708305 14286 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:28.718402 14286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Generated new TSK 0
I20260927 18:35:28.719477 14286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:35:28.780874 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:29.121186 14302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:29.121500 14302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:29.121656 14302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:29.134371 14302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:29.134528 14302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:29.134713 14302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:29.144905 14302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:29.146392 14302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:29.147732 14302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:29.162439 14308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:29.162489 14310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:29.163789 14302 server_base.cc:1034] running on GCE node
W20260927 18:35:29.166332 14307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:29.167243 14302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:29.168388 14302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:29.169653 14302 hybrid_clock.cc:648] HybridClock initialized: now 1790534129169575 us; error 62 us; skew 500 ppm
I20260927 18:35:29.170204 14302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:29 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:29.173851 14302 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:29.175597 14302 webserver.cc:460] Webserver started at http://127.0.97.65:38889/ using document root <none> and password file <none>
I20260927 18:35:29.176342 14302 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:29.176476 14302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:29.176821 14302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:29.179903 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "df1441681e97e3371fbc068bde90c7af"
server_key_iv: "83072b2bd29e83f9854c93a5c0c4e644"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.180701 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "df1441681e97e3371fbc068bde90c7af"
server_key_iv: "83072b2bd29e83f9854c93a5c0c4e644"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.186288 14302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:35:29.190057 14317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.191468 14302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:35:29.191654 14302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "df1441681e97e3371fbc068bde90c7af"
server_key_iv: "83072b2bd29e83f9854c93a5c0c4e644"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.191861 14302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:29.212898 14302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:29.217404 14302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:29.217710 14302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:29.218788 14302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:29.220135 14302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:29.220211 14302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.220328 14302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:29.220384 14302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.252941 14302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42777
I20260927 18:35:29.253075 14429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42777 every 8 connection(s)
I20260927 18:35:29.254433 14302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:29.255697 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14302
I20260927 18:35:29.255930 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:29.256747 389 external_mini_cluster.cc:1428] Setting key f53e6b4234bdc91d35962ca1f4baed85
Sep 27 18:35:29 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:29.280942 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:29.257612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54423 (local address 127.0.97.126:37019)
0927 18:35:29.258152 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:29.258157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:29.259314 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 18:35:29.262402 (+ 3088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:29.262407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:29.262477 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:29.262601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:29.264904 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.265726 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.268106 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.268866 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.269018 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:29.273193 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:29.273203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:29.273211 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:29.273259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:29.277154 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:29.278049 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:29.278057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:29.278063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:29.278226 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:29.278865 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:29.278870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:29.278873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:29.279190 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:29.279311 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:29.280436 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0927 18:35:29.280624 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
I20260927 18:35:29.282619 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:29.258256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:54423)
0927 18:35:29.259129 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:29.259187 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:29.262018 (+ 2831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:29.262891 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:29.262907 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:29.263750 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:29.264659 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:29.264692 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.265980 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.265988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:29.267929 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:29.267941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.269087 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.269091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:29.269290 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:29.270072 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:29.270132 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:29.272921 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:29.277310 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:29.277319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:29.277334 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:29.277920 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:29.278354 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:29.278357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:29.278359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:29.278716 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:29.279340 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:29.279350 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:29.279892 (+ 542us) client_negotiation.cc:769] Sending connection context
0927 18:35:29.280292 (+ 400us) client_negotiation.cc:240] Negotiation successful
0927 18:35:29.280991 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260927 18:35:29.284512 14430 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:29.284849 14430 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:29.285598 14430 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260927 18:35:29.287976 14241 ts_manager.cc:194] Registered new tserver with Master: 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65:42777)
I20260927 18:35:29.289443 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:54423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:35:29.317222 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:29.650471 14436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:29.650743 14436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:29.650840 14436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:29.658382 14436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:29.658492 14436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:29.658604 14436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:29.668164 14436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:29.669309 14436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:29.670521 14436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:29.683439 14444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:29.683606 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:29.684017 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:29.684356 14436 server_base.cc:1034] running on GCE node
I20260927 18:35:29.684958 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:29.685932 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:29.687150 14436 hybrid_clock.cc:648] HybridClock initialized: now 1790534129687066 us; error 74 us; skew 500 ppm
I20260927 18:35:29.687451 14436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:29 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:29.690634 14436 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:29.691996 14436 webserver.cc:460] Webserver started at http://127.0.97.66:44887/ using document root <none> and password file <none>
I20260927 18:35:29.692788 14436 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:29.692926 14436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:29.693210 14436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:29.696138 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "2d008a8831e748e793e85bcd5ac6f27e"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "9799488f349322ad5fed75c5bd670770"
server_key_iv: "60830fdf229ae3f561c61fef2f04caa3"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.696897 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "2d008a8831e748e793e85bcd5ac6f27e"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "9799488f349322ad5fed75c5bd670770"
server_key_iv: "60830fdf229ae3f561c61fef2f04caa3"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.701655 14436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:35:29.705425 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.706857 14436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:29.707077 14436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "2d008a8831e748e793e85bcd5ac6f27e"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "9799488f349322ad5fed75c5bd670770"
server_key_iv: "60830fdf229ae3f561c61fef2f04caa3"
server_key_version: "encryptionkey@0"
I20260927 18:35:29.707283 14436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:29.730564 14436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:29.734977 14436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:29.735301 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:29.736294 14436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:29.737668 14436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:29.737804 14436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.737962 14436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:29.738036 14436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:29.772018 14436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41311
I20260927 18:35:29.772114 14563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41311 every 8 connection(s)
I20260927 18:35:29.773306 14436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:29.780089 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14436
I20260927 18:35:29.780294 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:29.780826 389 external_mini_cluster.cc:1428] Setting key bdb362a51eb9088775c75fef974d2d5a
Sep 27 18:35:29 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:29.795498 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:29.775974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35783 (local address 127.0.97.126:37019)
0927 18:35:29.776212 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:29.776218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:29.777302 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 18:35:29.779179 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:29.779183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:29.779252 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:29.779354 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:29.781277 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.782287 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.784323 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.784740 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.784835 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:29.789005 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:29.789010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:29.789013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:29.789052 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:29.792258 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:29.793022 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:29.793028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:29.793031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:29.793213 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:29.793628 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:29.793634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:29.793637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:29.794016 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:29.794193 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:29.794909 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 18:35:29.795142 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:35:29.796427 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:29.776459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:35783)
0927 18:35:29.777129 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:29.777169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:29.778908 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:29.779558 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:29.779566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:29.780328 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:29.781096 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:29.781122 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.782444 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.782448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:29.784171 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:29.784189 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:29.784980 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:29.784991 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:29.785309 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:29.786112 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:29.786147 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:29.788785 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:29.792391 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:29.792399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:29.792415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:29.792901 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:29.793319 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:29.793324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:29.793327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:29.793511 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:29.794159 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:29.794167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:29.794532 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 18:35:29.794818 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 18:35:29.795340 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":250,"threads_started":1}
I20260927 18:35:29.798085 14564 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:29.798445 14564 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:29.799338 14564 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
I20260927 18:35:29.801044 14241 ts_manager.cc:194] Registered new tserver with Master: 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311)
I20260927 18:35:29.801930 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:35783
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:35:29.837286 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:30.161764 14569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:30.162088 14569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:30.162184 14569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:30.169512 14569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:30.169631 14569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:30.169749 14569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:30.179891 14569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:30.181104 14569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:30.182406 14569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:30.195794 14574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:30.196002 14575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:30.196367 14577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:30.196753 14569 server_base.cc:1034] running on GCE node
I20260927 18:35:30.197475 14569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:30.198482 14569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:30.199687 14569 hybrid_clock.cc:648] HybridClock initialized: now 1790534130199651 us; error 35 us; skew 500 ppm
I20260927 18:35:30.200016 14569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:30 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:30.207934 14569 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:30.209419 14569 webserver.cc:460] Webserver started at http://127.0.97.67:33607/ using document root <none> and password file <none>
I20260927 18:35:30.210285 14569 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:30.210388 14569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:30.210636 14569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:30.213618 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "3452295193f24451aea13162090328ff"
format_stamp: "Formatted at 2026-09-27 18:35:30 on dist-test-slave-n3p4"
server_key: "823f52a489358d6fe89d09ee18f77d77"
server_key_iv: "c0d2e8dfe8a8d434d3ca01e56319d6ca"
server_key_version: "encryptionkey@0"
I20260927 18:35:30.214412 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "3452295193f24451aea13162090328ff"
format_stamp: "Formatted at 2026-09-27 18:35:30 on dist-test-slave-n3p4"
server_key: "823f52a489358d6fe89d09ee18f77d77"
server_key_iv: "c0d2e8dfe8a8d434d3ca01e56319d6ca"
server_key_version: "encryptionkey@0"
I20260927 18:35:30.219484 14569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:35:30.222950 14584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:30.224357 14569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:35:30.224545 14569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "3452295193f24451aea13162090328ff"
format_stamp: "Formatted at 2026-09-27 18:35:30 on dist-test-slave-n3p4"
server_key: "823f52a489358d6fe89d09ee18f77d77"
server_key_iv: "c0d2e8dfe8a8d434d3ca01e56319d6ca"
server_key_version: "encryptionkey@0"
I20260927 18:35:30.224716 14569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:30.238559 14569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:30.242652 14569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:30.242938 14569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:30.243903 14569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:30.245229 14569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:30.245298 14569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:30.245440 14569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:30.245491 14569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:30.279707 14569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33341
I20260927 18:35:30.279743 14696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33341 every 8 connection(s)
I20260927 18:35:30.281086 14569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:30.289917 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14569
I20260927 18:35:30.290120 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:30.290571 389 external_mini_cluster.cc:1428] Setting key a815788ea31fa745c2b723c432dd575d
I20260927 18:35:30.293406 14430 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
Sep 27 18:35:30 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:30.302031 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.283566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60137 (local address 127.0.97.126:37019)
0927 18:35:30.283779 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.283785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.284937 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 18:35:30.286800 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.286804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.286934 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.287037 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:30.288641 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.289408 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.291307 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.291923 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.292076 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.296263 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:30.296273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:30.296276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:30.296307 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:30.299054 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.299738 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.299742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.299744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.299874 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.300281 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.300284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.300286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.300699 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:30.300812 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.301511 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.301684 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:35:30.302982 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.284078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:60137)
0927 18:35:30.284777 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.284819 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.286519 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.287227 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.287235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.287744 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:30.288456 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.288476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.289564 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.289568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.291137 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.291153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.292156 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.292162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.292460 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.293394 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:30.293428 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:30.296057 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:30.299190 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:30.299197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:30.299210 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:30.299617 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:30.299965 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:30.299968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:30.299970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:30.300172 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:30.300812 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:30.300817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:30.301157 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.301382 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.301773 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":268,"threads_started":1}
I20260927 18:35:30.304351 14697 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:30.304649 14697 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:30.305454 14697 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
I20260927 18:35:30.306860 14241 ts_manager.cc:194] Registered new tserver with Master: 3452295193f24451aea13162090328ff (127.0.97.67:33341)
I20260927 18:35:30.307605 14241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:60137
I20260927 18:35:30.318849 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:30.336122 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.323476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.0.97.126:37019)
0927 18:35:30.323668 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.323673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.324221 (+ 548us) server_negotiation.cc:407] Connection header received
0927 18:35:30.324361 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.324365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.324439 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.324523 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:30.326022 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.326731 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.328201 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.328629 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.328721 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.331270 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:30.331275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:30.331278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:30.331306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:30.333888 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.334507 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.334510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.334512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.334649 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.334948 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.334951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.334953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.335271 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:30.335450 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.335747 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.335895 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:35:30.340126 14241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:30.346935 14241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:30.372881 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.359298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33341 (local address 127.0.0.1:53138)
0927 18:35:30.360875 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.360942 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.361194 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.362399 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.362406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.362436 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:30.362922 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.362942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.365406 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.365414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.368510 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.368531 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.371843 (+ 3312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.371849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.372079 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.372160 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.372296 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.372493 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":1137,"threads_started":1}
I20260927 18:35:30.372959 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.359377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.97.67:33341)
0927 18:35:30.360461 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.360472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.361433 (+ 961us) server_negotiation.cc:407] Connection header received
0927 18:35:30.361563 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.361573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.361884 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.362096 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:30.363139 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.365157 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.368770 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.370710 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.370849 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.371656 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.372382 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.372591 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":725,"threads_started":1}
I20260927 18:35:30.377210 14632 tablet_service.cc:1467] Processing CreateTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df (DEFAULT_TABLE table=test-table [id=f1edb28a5edb4c7782e3f6828200b84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:30.378804 14632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f77fdb55f6c437999f53cbd7440b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:30.381362 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.360550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.97.66:41311)
0927 18:35:30.361182 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.361191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.372187 (+ 10996us) server_negotiation.cc:407] Connection header received
0927 18:35:30.372493 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.372503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.372761 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.372944 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:30.374279 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.376101 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.378404 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.380140 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.380305 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.380529 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.380793 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.380985 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
I20260927 18:35:30.381690 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.361445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50420)
0927 18:35:30.372302 (+ 10857us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.372322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.372452 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.373288 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.373293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.373318 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:30.373853 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.373868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.376181 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.376186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.378426 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.378441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.380720 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.380730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.380929 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.380964 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.381061 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.381222 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10679,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":9936,"threads_started":1}
I20260927 18:35:30.382501 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.359919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41311 (local address 127.0.0.1:48930)
0927 18:35:30.372040 (+ 12121us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.372068 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.372309 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.373424 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.373431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.373468 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:30.374084 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.374100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.376336 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.376343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.378211 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.378224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.380362 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.380370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.380567 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.380604 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.380705 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.380849 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11951,"mutex_wait_us":27,"spinlock_wait_cycles":896,"thread_start_us":11996,"threads_started":1}
I20260927 18:35:30.382535 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.361803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.0.97.65:42777)
0927 18:35:30.371350 (+ 9547us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.371361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.372588 (+ 1227us) server_negotiation.cc:407] Connection header received
0927 18:35:30.372694 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.372706 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.373025 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.373490 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:30.374169 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.376053 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.378586 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.380209 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.380328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.380549 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.382038 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.382225 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9378,"thread_start_us":9246,"threads_started":1}
I20260927 18:35:30.386163 14365 tablet_service.cc:1467] Processing CreateTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df (DEFAULT_TABLE table=test-table [id=f1edb28a5edb4c7782e3f6828200b84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:30.387406 14365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f77fdb55f6c437999f53cbd7440b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:30.387215 14499 tablet_service.cc:1467] Processing CreateTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df (DEFAULT_TABLE table=test-table [id=f1edb28a5edb4c7782e3f6828200b84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:30.388593 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f77fdb55f6c437999f53cbd7440b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:30.399302 14715 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Bootstrap starting.
I20260927 18:35:30.403892 14715 tablet_bootstrap.cc:654] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:30.404842 14716 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Bootstrap starting.
I20260927 18:35:30.405629 14715 log.cc:824] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:30.409576 14716 tablet_bootstrap.cc:654] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:30.410838 14718 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Bootstrap starting.
I20260927 18:35:30.411948 14716 log.cc:824] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:30.413100 14715 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: No bootstrap required, opened a new log
I20260927 18:35:30.413538 14715 ts_tablet_manager.cc:1397] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 18:35:30.414726 14718 tablet_bootstrap.cc:654] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:30.416029 14716 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: No bootstrap required, opened a new log
I20260927 18:35:30.416047 14718 log.cc:824] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:30.416325 14716 ts_tablet_manager.cc:1397] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 18:35:30.419334 14718 tablet_bootstrap.cc:492] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: No bootstrap required, opened a new log
I20260927 18:35:30.419615 14718 ts_tablet_manager.cc:1397] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 18:35:30.422724 14715 raft_consensus.cc:348] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.423113 14715 raft_consensus.cc:374] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:30.423228 14715 raft_consensus.cc:729] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3452295193f24451aea13162090328ff, State: Initialized, Role: FOLLOWER
I20260927 18:35:30.423882 14715 consensus_queue.cc:260] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.425028 14697 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
I20260927 18:35:30.425817 14715 ts_tablet_manager.cc:1428] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260927 18:35:30.428392 14716 raft_consensus.cc:348] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.428835 14716 raft_consensus.cc:374] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:30.428931 14716 raft_consensus.cc:729] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a164273c6dc4b5fa5dc2d6f226a78e8, State: Initialized, Role: FOLLOWER
I20260927 18:35:30.429556 14716 consensus_queue.cc:260] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.430255 14718 raft_consensus.cc:348] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.430701 14718 raft_consensus.cc:374] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:30.430797 14718 raft_consensus.cc:729] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d008a8831e748e793e85bcd5ac6f27e, State: Initialized, Role: FOLLOWER
I20260927 18:35:30.431484 14716 ts_tablet_manager.cc:1428] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 18:35:30.431483 14718 consensus_queue.cc:260] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.436079 14564 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
I20260927 18:35:30.436811 14718 ts_tablet_manager.cc:1428] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Time spent starting tablet: real 0.017s user 0.015s sys 0.003s
W20260927 18:35:30.506866 14431 tablet.cc:2369] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:30.512318 14722 raft_consensus.cc:482] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:30.512607 14722 raft_consensus.cc:504] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.514531 14722 leader_election.cc:288] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3452295193f24451aea13162090328ff (127.0.97.67:33341), 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311)
I20260927 18:35:30.521860 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.515644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56631 (local address 127.0.97.67:33341)
0927 18:35:30.515862 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.515871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.516246 (+ 375us) server_negotiation.cc:407] Connection header received
0927 18:35:30.516388 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.516392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.516493 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.516599 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:30.517166 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.518069 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.519882 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.520797 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.520936 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.521083 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.521420 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.521557 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:35:30.522039 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.515019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36213 (local address 127.0.97.66:41311)
0927 18:35:30.515237 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.515244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.515930 (+ 686us) server_negotiation.cc:407] Connection header received
0927 18:35:30.516259 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.516265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.516350 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.516435 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:30.517210 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.517960 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.519962 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.520898 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.521030 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.521115 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.521566 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.521711 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:35:30.522045 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.514905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33341 (local address 127.0.97.65:56631)
0927 18:35:30.516097 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.516111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.516261 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.516625 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.516628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.516659 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:30.516973 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.516985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.518203 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.518206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.519718 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.519732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.521039 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.521045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.521222 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.521267 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.521385 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.521506 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":4,"thread_start_us":524,"threads_started":1}
I20260927 18:35:30.522545 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.514782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41311 (local address 127.0.97.65:36213)
0927 18:35:30.515809 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:30.515832 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:30.516081 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:30.516481 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:30.516494 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:30.516522 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:30.517030 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.517041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.518089 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.518095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.519779 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:30.519820 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.521173 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.521179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:30.521370 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.521402 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:35:30.521530 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:35:30.521661 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":61,"spinlock_wait_cycles":1408,"thread_start_us":1088,"threads_started":1}
I20260927 18:35:30.523075 14652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f77fdb55f6c437999f53cbd7440b4df" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3452295193f24451aea13162090328ff" is_pre_election: true
I20260927 18:35:30.523540 14519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f77fdb55f6c437999f53cbd7440b4df" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d008a8831e748e793e85bcd5ac6f27e" is_pre_election: true
I20260927 18:35:30.523725 14652 raft_consensus.cc:2454] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 0.
I20260927 18:35:30.524080 14519 raft_consensus.cc:2454] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 0.
I20260927 18:35:30.524437 14320 leader_election.cc:302] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a164273c6dc4b5fa5dc2d6f226a78e8, 3452295193f24451aea13162090328ff; no voters:
I20260927 18:35:30.524857 14722 raft_consensus.cc:2781] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:30.525028 14722 raft_consensus.cc:482] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:30.525173 14722 raft_consensus.cc:3037] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:35:30.525506 14565 tablet.cc:2369] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:30.527611 14722 raft_consensus.cc:504] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.528246 14722 leader_election.cc:288] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3452295193f24451aea13162090328ff (127.0.97.67:33341), 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311)
I20260927 18:35:30.528702 14519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f77fdb55f6c437999f53cbd7440b4df" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d008a8831e748e793e85bcd5ac6f27e"
I20260927 18:35:30.528697 14652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f77fdb55f6c437999f53cbd7440b4df" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3452295193f24451aea13162090328ff"
I20260927 18:35:30.528873 14519 raft_consensus.cc:3037] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:30.528905 14652 raft_consensus.cc:3037] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:30.531189 14519 raft_consensus.cc:2454] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 1.
I20260927 18:35:30.531441 14652 raft_consensus.cc:2454] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 1.
I20260927 18:35:30.531567 14318 leader_election.cc:302] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a164273c6dc4b5fa5dc2d6f226a78e8, 2d008a8831e748e793e85bcd5ac6f27e; no voters:
I20260927 18:35:30.531910 14722 raft_consensus.cc:2781] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:30.532308 14722 raft_consensus.cc:686] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1a164273c6dc4b5fa5dc2d6f226a78e8, State: Running, Role: LEADER
W20260927 18:35:30.532681 14698 tablet.cc:2369] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:30.532819 14722 consensus_queue.cc:237] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } }
I20260927 18:35:30.537412 14241 catalog_manager.cc:5495] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } health_report { overall_health: UNKNOWN } } }
I20260927 18:35:30.565303 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.559940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.97.65:42777)
0927 18:35:30.560160 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.560165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.560185 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:35:30.560265 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.560268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.560339 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.560465 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:30.560817 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.561555 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.563104 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.563529 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.563626 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.564201 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:30.564691 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:30.564805 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.564940 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.565005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:35:30.574115 14519 raft_consensus.cc:1261] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Refusing update from remote peer 1a164273c6dc4b5fa5dc2d6f226a78e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:30.574105 14652 raft_consensus.cc:1261] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Refusing update from remote peer 1a164273c6dc4b5fa5dc2d6f226a78e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:30.575091 14726 consensus_queue.cc:1035] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:30.575453 14722 consensus_queue.cc:1035] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:30.587438 14731 mvcc.cc:205] Tried to move back new op lower bound from 7334027798821097472 to 7334027798666366976. Current Snapshot: MvccSnapshot[applied={T|T < 7334027798821097472}]
I20260927 18:35:30.588792 14729 mvcc.cc:205] Tried to move back new op lower bound from 7334027798821097472 to 7334027798666366976. Current Snapshot: MvccSnapshot[applied={T|T < 7334027798821097472}]
I20260927 18:35:30.603075 14241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:30.603386 14241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:30.607052 14241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Sending DeleteTablet for 3 replicas of tablet 7f77fdb55f6c437999f53cbd7440b4df
I20260927 18:35:30.608131 14632 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53138
I20260927 18:35:30.608167 14365 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50420
I20260927 18:35:30.608796 14499 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f77fdb55f6c437999f53cbd7440b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:48930
I20260927 18:35:30.609817 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14302
I20260927 18:35:30.620422 14740 tablet_replica.cc:330] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: stopping tablet replica
I20260927 18:35:30.620894 14740 raft_consensus.cc:2229] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:30.621371 14741 tablet_replica.cc:330] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: stopping tablet replica
I20260927 18:35:30.621536 14740 raft_consensus.cc:2258] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:30.622071 14741 raft_consensus.cc:2229] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:30.622355 14739 tablet_replica.cc:330] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: stopping tablet replica
I20260927 18:35:30.622725 14739 raft_consensus.cc:2229] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:35:30.622819 14741 raft_consensus.cc:2258] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:30.625314 14740 ts_tablet_manager.cc:1905] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:30.625402 14741 ts_tablet_manager.cc:1905] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:30.628093 14739 raft_consensus.cc:2258] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:30.630264 14739 ts_tablet_manager.cc:1905] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:30.633327 14741 ts_tablet_manager.cc:1918] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:30.633505 14741 log.cc:1196] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/7f77fdb55f6c437999f53cbd7440b4df
I20260927 18:35:30.634137 14741 ts_tablet_manager.cc:1939] T 7f77fdb55f6c437999f53cbd7440b4df P 2d008a8831e748e793e85bcd5ac6f27e: Deleting consensus metadata
I20260927 18:35:30.636179 14225 catalog_manager.cc:4841] TS 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311): tablet 7f77fdb55f6c437999f53cbd7440b4df (table test-table [id=f1edb28a5edb4c7782e3f6828200b84c]) successfully deleted
Sep 27 18:35:30 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:35:30.636895 14739 ts_tablet_manager.cc:1918] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:30.637046 14739 log.cc:1196] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/7f77fdb55f6c437999f53cbd7440b4df
I20260927 18:35:30.637557 14739 ts_tablet_manager.cc:1939] T 7f77fdb55f6c437999f53cbd7440b4df P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting consensus metadata
I20260927 18:35:30.642639 14740 ts_tablet_manager.cc:1918] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:30.642807 14740 log.cc:1196] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/7f77fdb55f6c437999f53cbd7440b4df
I20260927 18:35:30.643283 14740 ts_tablet_manager.cc:1939] T 7f77fdb55f6c437999f53cbd7440b4df P 3452295193f24451aea13162090328ff: Deleting consensus metadata
I20260927 18:35:30.645087 14227 catalog_manager.cc:4841] TS 3452295193f24451aea13162090328ff (127.0.97.67:33341): tablet 7f77fdb55f6c437999f53cbd7440b4df (table test-table [id=f1edb28a5edb4c7782e3f6828200b84c]) successfully deleted
I20260927 18:35:30.646723 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.611169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.97.65:42777)
0927 18:35:30.611419 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.611428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.623971 (+ 12543us) server_negotiation.cc:407] Connection header received
0927 18:35:30.624255 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.624261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.624362 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.624448 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:30.626796 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.628159 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.630580 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.631399 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.631517 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.638016 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:30.638031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:30.638056 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:30.638106 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:30.642670 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.643716 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.643726 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.643731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.643920 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.644438 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.644444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.644448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.644929 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:30.645244 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.646032 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.646327 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:35:30.646795 14227 catalog_manager.cc:4841] TS 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65:42777): tablet 7f77fdb55f6c437999f53cbd7440b4df (table test-table [id=f1edb28a5edb4c7782e3f6828200b84c]) successfully deleted
I20260927 18:35:30.648370 14425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:30.751859 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14302
I20260927 18:35:30.771607 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14436
Sep 27 18:35:30 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:35:30.789928 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.772519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.0.97.66:41311)
0927 18:35:30.772696 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.772700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.772719 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:35:30.772852 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.772855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.772924 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.773030 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:30.774799 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.775795 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.777923 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.778362 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.778461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.783666 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:30.783677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:30.783695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:30.783745 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:30.786889 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.787680 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.787687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.787690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.787812 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.788200 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.788207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.788210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.788622 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:30.788906 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.789256 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.789563 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:35:30.790694 14559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:30.882347 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14436
I20260927 18:35:30.900098 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14569
Sep 27 18:35:30 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:35:30.915560 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:30.900920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.97.67:33341)
0927 18:35:30.901121 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:30.901126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:30.901145 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:35:30.901342 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:30.901348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:30.901429 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:30.901532 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:30.902922 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.903665 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.904987 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:30.905392 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:30.905525 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:30.910183 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:30.910199 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:30.910215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:30.910256 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:30.912775 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.913429 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.913438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.913442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.913591 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:30.914037 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:30.914043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:30.914047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:30.914396 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:30.914700 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:30.915012 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 18:35:30.915236 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:35:30.916291 14692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:31.006639 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14569
I20260927 18:35:31.023664 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14209
I20260927 18:35:31.024914 14271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:31.137030 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14209
I20260927 18:35:31.158300 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--webserver_interface=127.0.97.126
--webserver_port=44171
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:31.492190 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:31.492476 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:31.492568 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:31.499953 14747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:31.500106 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:31.500159 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:31.500216 14747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:31.500262 14747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:31.510334 14747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:31.511690 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:31.512988 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:31.536870 14753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:31.536778 14754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:31.536815 14756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:31.537737 14747 server_base.cc:1034] running on GCE node
I20260927 18:35:31.538671 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:31.540275 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:31.541488 14747 hybrid_clock.cc:648] HybridClock initialized: now 1790534131541423 us; error 56 us; skew 500 ppm
I20260927 18:35:31.541999 14747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:31 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:31.545835 14747 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:31.547734 14747 webserver.cc:460] Webserver started at http://127.0.97.126:44171/ using document root <none> and password file <none>
I20260927 18:35:31.548535 14747 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:31.548692 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:31.555140 14747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:35:31.558377 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:31.559850 14747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:35:31.560050 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "fe165813fc0b4cdebc03a4ce94149ad3"
format_stamp: "Formatted at 2026-09-27 18:35:28 on dist-test-slave-n3p4"
server_key: "272b3bae4f2c3cf9c8c64fbfbc345a9e"
server_key_iv: "ff4d985243f6a16219b72aa2f58a19bb"
server_key_version: "encryptionkey@0"
I20260927 18:35:31.561178 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:31.596503 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:31.601001 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:31.601322 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:31.621037 14747 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37019
I20260927 18:35:31.621048 14814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37019 every 8 connection(s)
I20260927 18:35:31.622706 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:31.629618 14815 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:35:31.631644 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14747
I20260927 18:35:31.632448 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:42777
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=38889
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:31.633062 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap starting.
I20260927 18:35:31.652721 14815 log.cc:824] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:31.665143 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:35:31.665831 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap complete.
I20260927 18:35:31.675172 14815 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:31.676039 14815 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Initialized, Role: FOLLOWER
I20260927 18:35:31.676651 14815 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:31.676909 14815 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:31.677050 14815 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:31.677235 14815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:35:31.681120 14815 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:31.681671 14815 leader_election.cc:302] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe165813fc0b4cdebc03a4ce94149ad3; no voters:
I20260927 18:35:31.682346 14815 leader_election.cc:288] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:35:31.682492 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:35:31.683790 14819 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 LEADER]: Becoming Leader. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Running, Role: LEADER
I20260927 18:35:31.684430 14819 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:31.684824 14815 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:31.687125 14821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe165813fc0b4cdebc03a4ce94149ad3. Latest consensus state: current_term: 2 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:31.687526 14821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:31.687320 14820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:31.687702 14820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:31.699384 14826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:31.709653 14826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:31.711607 14826 catalog_manager.cc:1259] Loaded cluster ID: b6413f752c024fa799f515e98e9d5737
I20260927 18:35:31.711745 14826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:31.715873 14826 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:35:31.740995 14826 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:35:31.748404 14826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:31.749864 14826 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Loaded TSK: 0
I20260927 18:35:31.750996 14826 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:35:31.998452 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:31.998787 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:31.998962 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:32.006724 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:32.006901 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:32.007079 14817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:32.017273 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:42777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=38889
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:32.018731 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:32.020207 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:32.034248 14842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.034193 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.034945 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:32.035035 14817 server_base.cc:1034] running on GCE node
I20260927 18:35:32.035708 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:32.036743 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:32.037966 14817 hybrid_clock.cc:648] HybridClock initialized: now 1790534132037919 us; error 39 us; skew 500 ppm
I20260927 18:35:32.038311 14817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:32 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:32.041425 14817 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:32.043035 14817 webserver.cc:460] Webserver started at http://127.0.97.65:38889/ using document root <none> and password file <none>
I20260927 18:35:32.043712 14817 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:32.043850 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:32.049299 14817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:35:32.052119 14852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.053550 14817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:35:32.053722 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "df1441681e97e3371fbc068bde90c7af"
server_key_iv: "83072b2bd29e83f9854c93a5c0c4e644"
server_key_version: "encryptionkey@0"
I20260927 18:35:32.054755 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:32.068485 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:32.073083 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:32.073386 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:32.074503 14817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:32.075886 14817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:32.075969 14817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.076128 14817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:32.076205 14817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.112500 14817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42777
I20260927 18:35:32.112591 14964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42777 every 8 connection(s)
I20260927 18:35:32.114022 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:32.115876 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14817
I20260927 18:35:32.116557 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:41311
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=44887
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:32 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:32.145543 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:32.117172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:47677)
0927 18:35:32.118061 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:32.118107 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:32.119799 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:32.124985 (+ 5186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:32.124993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:32.125493 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:32.126142 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:32.126161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.128592 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.128596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:32.129887 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:32.129900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.131432 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.131437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:32.131718 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:32.132488 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:32.132510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:32.135241 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:32.140559 (+ 5318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:32.140567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:32.140580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:32.141109 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:32.141695 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:32.141698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:32.141700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:32.141962 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:32.142988 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:32.142994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:32.143325 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 18:35:32.143782 (+ 457us) client_negotiation.cc:240] Negotiation successful
0927 18:35:32.144282 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":368,"threads_started":1}
I20260927 18:35:32.145587 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:32.116951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47677 (local address 127.0.97.126:37019)
0927 18:35:32.117624 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:32.117644 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:32.118244 (+ 600us) server_negotiation.cc:407] Connection header received
0927 18:35:32.123890 (+ 5646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:32.123914 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:32.124451 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:32.124881 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:32.126368 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.128417 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.130028 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.131215 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.131401 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:32.135586 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:32.135599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:32.135622 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:32.135682 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:32.140373 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:32.141284 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:32.141296 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:32.141308 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:32.141547 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:32.142105 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:32.142113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:32.142118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:32.142836 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:32.143059 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:32.144023 (+ 964us) server_negotiation.cc:299] Negotiation successful
0927 18:35:32.144469 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":276,"threads_started":1}
I20260927 18:35:32.148729 14965 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:32.149048 14965 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:32.149959 14965 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
I20260927 18:35:32.152163 14780 ts_manager.cc:194] Registered new tserver with Master: 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65:42777)
I20260927 18:35:32.154017 14780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:47677
W20260927 18:35:32.449905 14969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:32.450184 14969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:32.450332 14969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:32.457489 14969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:32.457612 14969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:32.457842 14969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:32.467947 14969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:41311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=44887
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:32.469221 14969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:32.470534 14969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:32.483639 14975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.483843 14974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.484122 14977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:32.484757 14969 server_base.cc:1034] running on GCE node
I20260927 18:35:32.485394 14969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:32.486449 14969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:32.487676 14969 hybrid_clock.cc:648] HybridClock initialized: now 1790534132487621 us; error 52 us; skew 500 ppm
I20260927 18:35:32.487991 14969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:32 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:32.491109 14969 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:32.492519 14969 webserver.cc:460] Webserver started at http://127.0.97.66:44887/ using document root <none> and password file <none>
I20260927 18:35:32.493227 14969 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:32.493338 14969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:32.498695 14969 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:35:32.501392 14984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.502863 14969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:35:32.503039 14969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "2d008a8831e748e793e85bcd5ac6f27e"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "9799488f349322ad5fed75c5bd670770"
server_key_iv: "60830fdf229ae3f561c61fef2f04caa3"
server_key_version: "encryptionkey@0"
I20260927 18:35:32.503976 14969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:32.519366 14969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:32.523670 14969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:32.523943 14969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:32.524902 14969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:32.526255 14969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:32.526356 14969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.526504 14969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:32.526613 14969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.560307 14969 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41311
I20260927 18:35:32.560415 15096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41311 every 8 connection(s)
I20260927 18:35:32.561863 14969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:32.562381 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 14969
I20260927 18:35:32.563194 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:33341
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=33607
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:32 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:32.589186 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:32.564904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60127 (local address 127.0.97.126:37019)
0927 18:35:32.565135 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:32.565143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:32.569894 (+ 4751us) server_negotiation.cc:407] Connection header received
0927 18:35:32.571790 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:32.571796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:32.571894 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:32.572008 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:32.573711 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.574897 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.576644 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.577483 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.577640 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:32.581805 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:32.581814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:32.581822 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:32.581875 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:32.585742 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:32.586542 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:32.586551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:32.586556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:32.586728 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:32.587152 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:32.587160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:32.587166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:32.587574 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:32.587747 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:32.588427 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 18:35:32.588706 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:35:32.589951 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:32.565427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:60127)
0927 18:35:32.569680 (+ 4253us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:32.569722 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:32.571501 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:32.572319 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:32.572330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:32.572893 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:32.573514 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:32.573535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.575168 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.575172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:32.576484 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:32.576497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:32.577705 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:32.577710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:32.578134 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:32.578908 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:32.578931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:32.581541 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:32.585932 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:32.585940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:32.585954 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:32.586413 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:32.586847 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:32.586850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:32.586852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:32.587035 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:32.587700 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:32.587706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:32.588029 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 18:35:32.588273 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 18:35:32.588683 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3972,"thread_start_us":3798,"threads_started":1}
I20260927 18:35:32.591490 15097 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:32.591807 15097 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:32.592537 15097 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
I20260927 18:35:32.594059 14780 ts_manager.cc:194] Registered new tserver with Master: 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311)
I20260927 18:35:32.594854 14780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:60127
W20260927 18:35:32.903587 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:32.903882 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:32.904055 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:32.911453 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:32.911593 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:32.911777 15100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:32.921577 15100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=33607
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:32.922910 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:32.924269 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:32.937422 15105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.937536 15106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:32.937579 15108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:32.938459 15100 server_base.cc:1034] running on GCE node
I20260927 18:35:32.938982 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:32.939819 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:32.941018 15100 hybrid_clock.cc:648] HybridClock initialized: now 1790534132940975 us; error 27 us; skew 500 ppm
I20260927 18:35:32.941282 15100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:32 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:32.944301 15100 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:32.945749 15100 webserver.cc:460] Webserver started at http://127.0.97.67:33607/ using document root <none> and password file <none>
I20260927 18:35:32.946460 15100 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:32.946604 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:32.951839 15100 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:35:32.954537 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.955821 15100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:32.955991 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "3452295193f24451aea13162090328ff"
format_stamp: "Formatted at 2026-09-27 18:35:30 on dist-test-slave-n3p4"
server_key: "823f52a489358d6fe89d09ee18f77d77"
server_key_iv: "c0d2e8dfe8a8d434d3ca01e56319d6ca"
server_key_version: "encryptionkey@0"
I20260927 18:35:32.956960 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:32.980943 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:32.985157 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:32.985424 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:32.986544 15100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:32.987876 15100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:32.988029 15100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:32.988154 15100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:32.988240 15100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:33.022991 15100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33341
I20260927 18:35:33.023090 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33341 every 8 connection(s)
I20260927 18:35:33.024691 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:33.029537 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15100
Sep 27 18:35:33 dist-test-slave-n3p4 krb5kdc[14196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:33.048360 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.030773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.0.97.126:37019)
0927 18:35:33.031554 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.031560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.031578 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:35:33.031614 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.031618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.031680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.031768 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:33.033755 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.035098 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.037077 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.037702 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.037897 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.041114 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:33.041122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:33.041129 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:33.041177 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:33.045435 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.046193 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.046196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.046198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.046313 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.046734 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.046738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.046740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.047156 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:33.047327 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.047703 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.047915 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":670,"thread_start_us":549,"threads_started":1}
I20260927 18:35:33.049373 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.027610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45159 (local address 127.0.97.126:37019)
0927 18:35:33.027814 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.027819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.028744 (+ 925us) server_negotiation.cc:407] Connection header received
0927 18:35:33.031503 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.031508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.031612 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.031728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:33.033887 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.035098 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.037330 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.038098 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.038196 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.042351 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:33.042358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:33.042363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:33.042408 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:33.046020 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.046881 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.046886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.046889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.047020 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.047478 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.047481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.047483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.047770 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:33.047900 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.048873 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.049065 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":65}
I20260927 18:35:33.050361 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.027852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:45159)
0927 18:35:33.028565 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.028614 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.031130 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.032088 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.032100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.032847 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:33.033681 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.033709 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.035267 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.035272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.037204 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.037216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.038330 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.038341 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.038609 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.039246 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:33.039277 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:33.042141 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:33.046144 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:33.046156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:33.046171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:33.046763 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:33.047125 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:33.047128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:33.047130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:33.047362 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:33.047910 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:33.047918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:33.048279 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.048551 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.049097 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260927 18:35:33.051657 15228 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37019
I20260927 18:35:33.051954 15228 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:33.052623 15228 heartbeater.cc:511] Master 127.0.97.126:37019 requested a full tablet report, sending...
I20260927 18:35:33.054016 14780 ts_manager.cc:194] Registered new tserver with Master: 3452295193f24451aea13162090328ff (127.0.97.67:33341)
I20260927 18:35:33.054797 14780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:45159
I20260927 18:35:33.063006 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:33.070530 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.063943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.0.97.126:37019)
0927 18:35:33.064138 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.064147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.064593 (+ 446us) server_negotiation.cc:407] Connection header received
0927 18:35:33.064795 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.064798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.064954 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.065077 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:33.065710 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.066565 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.068342 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.068746 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.068842 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.069336 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:33.069920 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:33.070059 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.070178 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.070244 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:35:33.071441 14780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:33.078814 14780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:33.101501 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.093197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.97.65:42777)
0927 18:35:33.094366 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.094377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.094402 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:35:33.094506 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.094513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.094757 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.094951 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:33.095476 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096840 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.098914 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.100385 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.100530 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.100751 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.100999 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.101162 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":867,"threads_started":1}
I20260927 18:35:33.101634 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.092701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50448)
0927 18:35:33.093687 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.093725 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.093860 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.094938 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.094942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.094965 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:33.095314 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.095322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096982 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.096989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.098780 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.098791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.100574 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.100580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.100758 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.100827 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.100960 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.101154 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":2,"thread_start_us":709,"threads_started":1}
I20260927 18:35:33.102747 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.091342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.0.97.67:33341)
0927 18:35:33.092151 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.092160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.092732 (+ 572us) server_negotiation.cc:407] Connection header received
0927 18:35:33.093063 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.093072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.093334 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.093499 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:33.094656 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096381 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.099567 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.101020 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.101146 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.101367 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.102287 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.102441 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":1793,"threads_started":1}
I20260927 18:35:33.102902 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.090797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33341 (local address 127.0.0.1:53142)
0927 18:35:33.092612 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.092628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.092849 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.093950 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.093960 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.094015 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:33.094462 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.094473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096562 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.096566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.099371 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.099386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.101583 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.101589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.102028 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.102132 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.102224 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.102459 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"mutex_wait_us":258,"thread_start_us":1008,"threads_started":1}
I20260927 18:35:33.103475 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.091042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.97.66:41311)
0927 18:35:33.091697 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.091710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.092463 (+ 753us) server_negotiation.cc:407] Connection header received
0927 18:35:33.092905 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.092913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.093181 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.093371 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:33.094820 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096381 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.099555 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.101024 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.101146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.101330 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.103006 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.103189 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":322,"threads_started":1}
I20260927 18:35:33.104487 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.090564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41311 (local address 127.0.0.1:48958)
0927 18:35:33.092286 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.092343 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.092644 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.093947 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.093957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.094195 (+ 238us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:33.094637 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.094651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.096557 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.096562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.099371 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.099386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.101570 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.101578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.102070 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.102136 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.102224 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.102459 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":182,"thread_start_us":1750,"threads_started":1}
I20260927 18:35:33.105831 14900 tablet_service.cc:1467] Processing CreateTablet for tablet 71b2d6ddf98244048798da8ff9f4584c (DEFAULT_TABLE table=test-table [id=e5209cd8b0a1444d8554c54607616f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:33.107322 14900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b2d6ddf98244048798da8ff9f4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:33.106628 15163 tablet_service.cc:1467] Processing CreateTablet for tablet 71b2d6ddf98244048798da8ff9f4584c (DEFAULT_TABLE table=test-table [id=e5209cd8b0a1444d8554c54607616f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:33.108721 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b2d6ddf98244048798da8ff9f4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:33.108219 15032 tablet_service.cc:1467] Processing CreateTablet for tablet 71b2d6ddf98244048798da8ff9f4584c (DEFAULT_TABLE table=test-table [id=e5209cd8b0a1444d8554c54607616f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:33.109606 15032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b2d6ddf98244048798da8ff9f4584c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:33.124621 15244 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Bootstrap starting.
I20260927 18:35:33.127060 15246 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Bootstrap starting.
I20260927 18:35:33.128319 15244 tablet_bootstrap.cc:654] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:33.129344 15244 log.cc:824] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:33.130146 15245 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Bootstrap starting.
I20260927 18:35:33.131050 15246 tablet_bootstrap.cc:654] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:33.132323 15246 log.cc:824] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:33.133538 15244 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: No bootstrap required, opened a new log
I20260927 18:35:33.133855 15244 ts_tablet_manager.cc:1397] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:35:33.134191 15245 tablet_bootstrap.cc:654] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:33.135102 15245 log.cc:824] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:33.135501 15246 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: No bootstrap required, opened a new log
I20260927 18:35:33.135764 15246 ts_tablet_manager.cc:1397] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:35:33.137634 15245 tablet_bootstrap.cc:492] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: No bootstrap required, opened a new log
I20260927 18:35:33.137933 15245 ts_tablet_manager.cc:1397] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260927 18:35:33.144558 15244 raft_consensus.cc:348] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.145009 15244 raft_consensus.cc:374] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:33.145119 15244 raft_consensus.cc:729] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a164273c6dc4b5fa5dc2d6f226a78e8, State: Initialized, Role: FOLLOWER
I20260927 18:35:33.145089 15245 raft_consensus.cc:348] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.145547 15245 raft_consensus.cc:374] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:33.145651 15245 raft_consensus.cc:729] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3452295193f24451aea13162090328ff, State: Initialized, Role: FOLLOWER
I20260927 18:35:33.145810 15244 consensus_queue.cc:260] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.146030 15246 raft_consensus.cc:348] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.146486 15246 raft_consensus.cc:374] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:33.146337 15245 consensus_queue.cc:260] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.146610 15246 raft_consensus.cc:729] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d008a8831e748e793e85bcd5ac6f27e, State: Initialized, Role: FOLLOWER
I20260927 18:35:33.147255 14965 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
I20260927 18:35:33.147173 15246 consensus_queue.cc:260] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.147876 15244 ts_tablet_manager.cc:1428] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260927 18:35:33.150316 15246 ts_tablet_manager.cc:1428] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260927 18:35:33.150491 15228 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
I20260927 18:35:33.150731 15097 heartbeater.cc:503] Master 127.0.97.126:37019 was elected leader, sending a full tablet report...
I20260927 18:35:33.152609 15245 ts_tablet_manager.cc:1428] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260927 18:35:33.235323 15250 raft_consensus.cc:482] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:33.235664 15250 raft_consensus.cc:504] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.237512 15250 leader_election.cc:288] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311), 3452295193f24451aea13162090328ff (127.0.97.67:33341)
I20260927 18:35:33.244588 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.238091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60739 (local address 127.0.97.67:33341)
0927 18:35:33.238416 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.238421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.238975 (+ 554us) server_negotiation.cc:407] Connection header received
0927 18:35:33.239155 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.239159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.239223 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.239299 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:33.239946 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.240722 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.242742 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.243572 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.243676 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.243785 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.244132 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.244292 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260927 18:35:33.244678 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.237946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:50677 (local address 127.0.97.66:41311)
0927 18:35:33.238206 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.238215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.238910 (+ 695us) server_negotiation.cc:407] Connection header received
0927 18:35:33.239183 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.239187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.239246 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.239350 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:33.240021 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.240703 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.242832 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.243665 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.243812 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.243905 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.244290 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.244440 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:35:33.244678 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.237979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33341 (local address 127.0.97.65:60739)
0927 18:35:33.238835 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.238849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.239006 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.239338 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.239343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.239366 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:33.239781 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.239794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.240872 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.240876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.242587 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.242599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.243793 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.243800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.243974 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.243999 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.244077 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.244158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":58,"thread_start_us":593,"threads_started":1}
I20260927 18:35:33.245025 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.237717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41311 (local address 127.0.97.65:50677)
0927 18:35:33.238761 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.238787 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:33.238970 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:33.239389 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:33.239394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:33.239423 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:33.239851 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.239882 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.240845 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.240850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.242635 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:33.242670 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.243908 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.243913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:33.244087 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.244124 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:35:33.244239 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:35:33.244361 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":62,"thread_start_us":597,"threads_started":1}
I20260927 18:35:33.245674 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b2d6ddf98244048798da8ff9f4584c" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3452295193f24451aea13162090328ff" is_pre_election: true
I20260927 18:35:33.246284 15183 raft_consensus.cc:2454] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 0.
I20260927 18:35:33.246747 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b2d6ddf98244048798da8ff9f4584c" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d008a8831e748e793e85bcd5ac6f27e" is_pre_election: true
I20260927 18:35:33.246932 14855 leader_election.cc:302] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a164273c6dc4b5fa5dc2d6f226a78e8, 3452295193f24451aea13162090328ff; no voters:
I20260927 18:35:33.247254 15052 raft_consensus.cc:2454] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 0.
I20260927 18:35:33.247364 15250 raft_consensus.cc:2781] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:33.247536 15250 raft_consensus.cc:482] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:33.247676 15250 raft_consensus.cc:3037] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:33.250026 15250 raft_consensus.cc:504] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.250694 15250 leader_election.cc:288] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 election: Requested vote from peers 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311), 3452295193f24451aea13162090328ff (127.0.97.67:33341)
I20260927 18:35:33.251070 15052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b2d6ddf98244048798da8ff9f4584c" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d008a8831e748e793e85bcd5ac6f27e"
I20260927 18:35:33.251111 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b2d6ddf98244048798da8ff9f4584c" candidate_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3452295193f24451aea13162090328ff"
I20260927 18:35:33.251282 15052 raft_consensus.cc:3037] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:33.251307 15183 raft_consensus.cc:3037] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:33.253479 15183 raft_consensus.cc:2454] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 1.
I20260927 18:35:33.253626 15052 raft_consensus.cc:2454] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a164273c6dc4b5fa5dc2d6f226a78e8 in term 1.
I20260927 18:35:33.253892 14855 leader_election.cc:302] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a164273c6dc4b5fa5dc2d6f226a78e8, 3452295193f24451aea13162090328ff; no voters:
I20260927 18:35:33.254189 15250 raft_consensus.cc:2781] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:33.254765 15250 raft_consensus.cc:686] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1a164273c6dc4b5fa5dc2d6f226a78e8, State: Running, Role: LEADER
I20260927 18:35:33.255359 15250 consensus_queue.cc:237] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } }
I20260927 18:35:33.260527 14780 catalog_manager.cc:5495] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42777 } health_report { overall_health: HEALTHY } } }
W20260927 18:35:33.276458 15229 tablet.cc:2369] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:33.305652 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.300432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.0.97.65:42777)
0927 18:35:33.300644 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.300648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.300666 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:35:33.300804 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.300808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.300896 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.301019 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:33.301379 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.302205 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.303601 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.304008 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.304108 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.304562 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:33.305120 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:33.305228 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.305380 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.305444 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:35:33.313621 15183 raft_consensus.cc:1261] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Refusing update from remote peer 1a164273c6dc4b5fa5dc2d6f226a78e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:33.313635 15052 raft_consensus.cc:1261] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Refusing update from remote peer 1a164273c6dc4b5fa5dc2d6f226a78e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 18:35:33.314277 15098 tablet.cc:2369] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:33.314729 15255 consensus_queue.cc:1035] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3452295193f24451aea13162090328ff" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:33.315105 15250 consensus_queue.cc:1035] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d008a8831e748e793e85bcd5ac6f27e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:33.325932 15257 mvcc.cc:205] Tried to move back new op lower bound from 7334027810043412480 to 7334027809818615808. Current Snapshot: MvccSnapshot[applied={T|T < 7334027810043412480}]
I20260927 18:35:33.330644 15259 mvcc.cc:205] Tried to move back new op lower bound from 7334027810043412480 to 7334027809818615808. Current Snapshot: MvccSnapshot[applied={T|T < 7334027810043412480}]
I20260927 18:35:33.339635 14780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:33.339884 14780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:33.343168 14780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Sending DeleteTablet for 3 replicas of tablet 71b2d6ddf98244048798da8ff9f4584c
I20260927 18:35:33.344369 15163 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b2d6ddf98244048798da8ff9f4584c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53142
I20260927 18:35:33.344467 15032 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b2d6ddf98244048798da8ff9f4584c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:48958
I20260927 18:35:33.344564 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14817
I20260927 18:35:33.344496 14900 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b2d6ddf98244048798da8ff9f4584c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50448
I20260927 18:35:33.345993 15267 tablet_replica.cc:330] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: stopping tablet replica
I20260927 18:35:33.348592 15267 raft_consensus.cc:2229] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:33.349066 15267 raft_consensus.cc:2258] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:33.358306 15269 tablet_replica.cc:330] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: stopping tablet replica
I20260927 18:35:33.358580 15267 ts_tablet_manager.cc:1905] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:33.358770 15269 raft_consensus.cc:2229] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:35:33.359627 15269 raft_consensus.cc:2258] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:33.359974 15268 tablet_replica.cc:330] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: stopping tablet replica
I20260927 18:35:33.360409 15268 raft_consensus.cc:2229] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:33.362280 15269 ts_tablet_manager.cc:1905] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:33.362972 15268 raft_consensus.cc:2258] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:33.365242 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.345651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.97.65:42777)
0927 18:35:33.345870 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.345876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.345893 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:35:33.345957 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.345965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.346076 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.346180 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:33.348259 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.349479 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.351939 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.352627 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.352752 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.356940 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:33.356955 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:33.356972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:33.357014 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:33.361987 (+ 4973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.362756 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.362764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.362767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.362941 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.363340 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.363345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.363348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.363810 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:33.364158 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.364507 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.364719 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:35:33.365720 15268 ts_tablet_manager.cc:1905] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:33.365896 14960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:33.370818 15267 ts_tablet_manager.cc:1918] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:33.371091 15267 log.cc:1196] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/71b2d6ddf98244048798da8ff9f4584c
I20260927 18:35:33.371634 15267 ts_tablet_manager.cc:1939] T 71b2d6ddf98244048798da8ff9f4584c P 3452295193f24451aea13162090328ff: Deleting consensus metadata
I20260927 18:35:33.372756 15268 ts_tablet_manager.cc:1918] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:33.372907 15268 log.cc:1196] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/71b2d6ddf98244048798da8ff9f4584c
I20260927 18:35:33.373418 15268 ts_tablet_manager.cc:1939] T 71b2d6ddf98244048798da8ff9f4584c P 2d008a8831e748e793e85bcd5ac6f27e: Deleting consensus metadata
I20260927 18:35:33.373530 14766 catalog_manager.cc:4841] TS 3452295193f24451aea13162090328ff (127.0.97.67:33341): tablet 71b2d6ddf98244048798da8ff9f4584c (table test-table [id=e5209cd8b0a1444d8554c54607616f2f]) successfully deleted
I20260927 18:35:33.375290 14764 catalog_manager.cc:4841] TS 2d008a8831e748e793e85bcd5ac6f27e (127.0.97.66:41311): tablet 71b2d6ddf98244048798da8ff9f4584c (table test-table [id=e5209cd8b0a1444d8554c54607616f2f]) successfully deleted
I20260927 18:35:33.470337 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14817
I20260927 18:35:33.488456 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14969
W20260927 18:35:33.488423 14766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:42777 (error 108)
I20260927 18:35:33.490368 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.489880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50474)
0927 18:35:33.490084 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.490207 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 18:35:33.490785 14766 catalog_manager.cc:4584] TS 1a164273c6dc4b5fa5dc2d6f226a78e8 (127.0.97.65:42777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71b2d6ddf98244048798da8ff9f4584c: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
I20260927 18:35:33.503137 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.489413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.0.97.66:41311)
0927 18:35:33.489618 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.489624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.489642 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:35:33.489765 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.489769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.489913 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.490047 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:33.491676 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.492439 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.494107 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.494512 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.494630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.497208 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:33.497223 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:33.497240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:33.497278 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:33.500527 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.501203 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.501209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.501212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.501333 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.501659 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.501662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.501664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.502135 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:33.502349 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.502711 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.502866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 18:35:33.503854 15092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:33.544381 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.543830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50482)
0927 18:35:33.544062 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.544175 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:35:33.545454 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.545061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50488)
0927 18:35:33.545222 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.545304 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:35:33.594056 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14969
I20260927 18:35:33.606112 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.605616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50492)
0927 18:35:33.605876 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.605964 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:35:33.607075 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.606744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50500)
0927 18:35:33.606884 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.606958 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 18:35:33.611774 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15100
I20260927 18:35:33.626153 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.612531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.0.97.67:33341)
0927 18:35:33.612719 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:33.612723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:33.612742 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:35:33.612911 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:33.612915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:33.612980 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:33.613066 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:33.614524 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.615294 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.616661 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:33.617063 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:33.617159 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:33.619899 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:33.619912 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:33.619929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:33.619960 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:33.623370 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.624148 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.624173 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.624175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.624314 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:33.624691 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:33.624694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:33.624696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:33.625053 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:33.625259 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:33.625573 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 18:35:33.625754 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:35:33.626832 15223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:33.687783 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.687306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50502)
0927 18:35:33.687516 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.687612 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:35:33.688812 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:33.688449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42777 (local address 127.0.0.1:50506)
0927 18:35:33.688636 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:33.688712 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 18:35:33.715467 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15100
I20260927 18:35:33.732409 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 14747
I20260927 18:35:33.733592 14810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:33.850466 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 14747
I20260927 18:35:33.872223 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--webserver_interface=127.0.97.126
--webserver_port=44171
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:34.194729 15275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:34.195039 15275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:34.195204 15275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:34.202435 15275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:34.202591 15275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:34.202698 15275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:34.202802 15275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:34.202888 15275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:34.212077 15275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:34.213326 15275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:34.214603 15275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:34.226816 15280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:34.226858 15281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:34.226811 15283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:34.227866 15275 server_base.cc:1034] running on GCE node
I20260927 18:35:34.228590 15275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:34.230000 15275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:34.231210 15275 hybrid_clock.cc:648] HybridClock initialized: now 1790534134231136 us; error 57 us; skew 500 ppm
I20260927 18:35:34.231534 15275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:34 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:34.234654 15275 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:34.236137 15275 webserver.cc:460] Webserver started at http://127.0.97.126:44171/ using document root <none> and password file <none>
I20260927 18:35:34.236789 15275 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:34.236918 15275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:34.242323 15275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:35:34.244976 15290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:34.246388 15275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:34.246554 15275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "fe165813fc0b4cdebc03a4ce94149ad3"
format_stamp: "Formatted at 2026-09-27 18:35:28 on dist-test-slave-n3p4"
server_key: "272b3bae4f2c3cf9c8c64fbfbc345a9e"
server_key_iv: "ff4d985243f6a16219b72aa2f58a19bb"
server_key_version: "encryptionkey@0"
I20260927 18:35:34.247476 15275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:34.272579 15275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:34.276973 15275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:34.277266 15275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:34.295533 15275 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37019
I20260927 18:35:34.295569 15341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37019 every 8 connection(s)
I20260927 18:35:34.297035 15275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:34.303910 15342 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:35:34.304198 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15275
I20260927 18:35:34.305178 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:42777
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=38889
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:34.308862 15342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap starting.
I20260927 18:35:34.330583 15342 log.cc:824] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:34.346398 15342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:35:34.346907 15342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3: Bootstrap complete.
I20260927 18:35:34.356616 15342 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:34.357621 15342 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Initialized, Role: FOLLOWER
I20260927 18:35:34.358326 15342 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:34.358575 15342 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:34.358700 15342 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:34.358881 15342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:35:34.362668 15342 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:34.363210 15342 leader_election.cc:302] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe165813fc0b4cdebc03a4ce94149ad3; no voters:
I20260927 18:35:34.363900 15342 leader_election.cc:288] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:35:34.364053 15346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:35:34.364794 15346 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [term 3 LEADER]: Becoming Leader. State: Replica: fe165813fc0b4cdebc03a4ce94149ad3, State: Running, Role: LEADER
I20260927 18:35:34.365401 15346 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } }
I20260927 18:35:34.366073 15342 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:34.368302 15347 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe165813fc0b4cdebc03a4ce94149ad3. Latest consensus state: current_term: 3 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:34.368337 15348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe165813fc0b4cdebc03a4ce94149ad3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37019 } } }
I20260927 18:35:34.368731 15347 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:34.368762 15348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe165813fc0b4cdebc03a4ce94149ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:34.383206 15351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:34.392339 15351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:34.394187 15351 catalog_manager.cc:1259] Loaded cluster ID: b6413f752c024fa799f515e98e9d5737
I20260927 18:35:34.394309 15351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:35:34.396584 15351 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f99392a89ad google::LogMessage::Fail() at ??:0
@ 0x7f99392ab88a google::LogMessage::SendToLog() at ??:0
@ 0x7f99392a84de google::LogMessage::Flush() at ??:0
@ 0x7f99392a9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f994ecdb512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f994ecd8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f994ed60610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f994ed60439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f993bef4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f993bef791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f993bef7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f993bed2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f993f1ac6db start_thread at ??:0
@ 0x7f993814261f clone at ??:0
W20260927 18:35:34.735351 15344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:34.735643 15344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:34.735743 15344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:34.743319 15344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:34.743453 15344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:34.743574 15344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:34.753450 15344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:42777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=38889
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:34.754772 15344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:34.756088 15344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:34.769434 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:34.769543 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:34.769582 15386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:34.771102 15344 server_base.cc:1034] running on GCE node
I20260927 18:35:34.771713 15344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:34.772645 15344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:34.773910 15344 hybrid_clock.cc:648] HybridClock initialized: now 1790534134773866 us; error 37 us; skew 500 ppm
I20260927 18:35:34.774279 15344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:34 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:34.777493 15344 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:34.779016 15344 webserver.cc:460] Webserver started at http://127.0.97.65:38889/ using document root <none> and password file <none>
I20260927 18:35:34.779740 15344 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:34.779886 15344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:34.785184 15344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:34.788266 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:34.789565 15344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:34.789744 15344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "1a164273c6dc4b5fa5dc2d6f226a78e8"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "df1441681e97e3371fbc068bde90c7af"
server_key_iv: "83072b2bd29e83f9854c93a5c0c4e644"
server_key_version: "encryptionkey@0"
I20260927 18:35:34.790812 15344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:34.817394 15344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:34.822016 15344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:34.822311 15344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:34.823335 15344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:34.825237 15400 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 18:35:34.830720 15400 ts_tablet_manager.cc:1852] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 18:35:34.831691 15400 ts_tablet_manager.cc:1905] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:34.840327 15400 ts_tablet_manager.cc:1918] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:34.840552 15400 log.cc:1196] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/71b2d6ddf98244048798da8ff9f4584c
I20260927 18:35:34.841044 15400 ts_tablet_manager.cc:1939] T 71b2d6ddf98244048798da8ff9f4584c P 1a164273c6dc4b5fa5dc2d6f226a78e8: Deleting consensus metadata
I20260927 18:35:34.841441 15344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 18:35:34.841580 15344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260927 18:35:34.841737 15344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:34.841847 15344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:34.876746 15344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42777
I20260927 18:35:34.876848 15506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42777 every 8 connection(s)
I20260927 18:35:34.878129 15344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:34.882083 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15344
I20260927 18:35:34.882886 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:41311
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=44887
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:34.883584 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:34.881134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:54029)
0927 18:35:34.882084 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:34.882376 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":307,"threads_started":1}
W20260927 18:35:34.884299 15507 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:37019: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
I20260927 18:35:34.885253 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:34.884818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:55307)
0927 18:35:34.885003 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:34.885105 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:35:34.886298 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:34.885871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:33773)
0927 18:35:34.886039 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:34.886137 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:35:34.886536 15507 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:35:35.205250 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:35.205526 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:35.205695 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:35.212797 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:35.212896 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:35.213083 15510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:35.222854 15510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:41311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=44887
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:35.224121 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:35.225442 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:35.238533 15516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:35.238730 15518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:35.238744 15515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:35.239687 15510 server_base.cc:1034] running on GCE node
I20260927 18:35:35.240265 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:35.241151 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:35.242338 15510 hybrid_clock.cc:648] HybridClock initialized: now 1790534135242281 us; error 53 us; skew 500 ppm
I20260927 18:35:35.242604 15510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:35 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:35.251363 15510 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:35.252818 15510 webserver.cc:460] Webserver started at http://127.0.97.66:44887/ using document root <none> and password file <none>
I20260927 18:35:35.253435 15510 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:35.253543 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:35.258826 15510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:35:35.261390 15525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.262835 15510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:35:35.262969 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "2d008a8831e748e793e85bcd5ac6f27e"
format_stamp: "Formatted at 2026-09-27 18:35:29 on dist-test-slave-n3p4"
server_key: "9799488f349322ad5fed75c5bd670770"
server_key_iv: "60830fdf229ae3f561c61fef2f04caa3"
server_key_version: "encryptionkey@0"
I20260927 18:35:35.263887 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:35.279018 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:35.283278 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:35.283506 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:35.284397 15510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:35.285633 15510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:35.285748 15510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.285883 15510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:35.285976 15510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.319527 15510 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41311
I20260927 18:35:35.319614 15637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41311 every 8 connection(s)
I20260927 18:35:35.321072 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:35.325646 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.323900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:39443)
0927 18:35:35.324611 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.324810 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260927 18:35:35.325639 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15510
W20260927 18:35:35.326472 15638 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:37019: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
I20260927 18:35:35.326449 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:33341
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:37019
--webserver_port=33607
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:35:35.327322 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.326962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:50137)
0927 18:35:35.327142 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.327223 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:35:35.328176 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.327861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:53305)
0927 18:35:35.328002 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.328078 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:35:35.328421 15638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:35:35.655831 15641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:35.656116 15641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:35.656216 15641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:35.663405 15641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:35.663524 15641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:35.663645 15641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:35.673401 15641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=33607
--tserver_master_addrs=127.0.97.126:37019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:35.674726 15641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:35.675981 15641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:35.689611 15649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:35.689509 15646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:35.689658 15647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:35.690665 15641 server_base.cc:1034] running on GCE node
I20260927 18:35:35.691262 15641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:35.692189 15641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:35.693441 15641 hybrid_clock.cc:648] HybridClock initialized: now 1790534135693380 us; error 51 us; skew 500 ppm
I20260927 18:35:35.693758 15641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:35 dist-test-slave-n3p4 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:35.697144 15641 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:35.698519 15641 webserver.cc:460] Webserver started at http://127.0.97.67:33607/ using document root <none> and password file <none>
I20260927 18:35:35.699188 15641 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:35.699350 15641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:35.704555 15641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:35:35.707222 15656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.708840 15641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:35.709013 15641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "3452295193f24451aea13162090328ff"
format_stamp: "Formatted at 2026-09-27 18:35:30 on dist-test-slave-n3p4"
server_key: "823f52a489358d6fe89d09ee18f77d77"
server_key_iv: "c0d2e8dfe8a8d434d3ca01e56319d6ca"
server_key_version: "encryptionkey@0"
I20260927 18:35:35.710021 15641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:35.737372 15641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:35.741964 15641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:35.742264 15641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:35.743311 15641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:35.744668 15641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:35.744753 15641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.744894 15641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:35.744990 15641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:35.778811 15641 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33341
I20260927 18:35:35.778879 15768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33341 every 8 connection(s)
I20260927 18:35:35.780196 15641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:35.785171 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.783335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:55709)
0927 18:35:35.784087 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.784298 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
W20260927 18:35:35.785923 15769 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:37019 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:37019: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
I20260927 18:35:35.786739 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.786436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:49567)
0927 18:35:35.786566 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.786630 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 18:35:35.787539 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.787265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:47613)
0927 18:35:35.787375 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.787441 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:35:35.787876 15769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:35:35.788748 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15641
I20260927 18:35:35.887933 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:35.887112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:42277)
0927 18:35:35.887582 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:35.887673 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260927 18:35:36.329761 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:36.328982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:53467)
0927 18:35:36.329420 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:36.329520 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":213,"threads_started":1}
I20260927 18:35:36.789377 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:36.788465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:55093)
0927 18:35:36.789019 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:36.789123 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":255,"threads_started":1}
I20260927 18:35:36.889475 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:36.888828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:54111)
0927 18:35:36.889238 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:36.889314 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260927 18:35:37.331413 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:37.330626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:57065)
0927 18:35:37.331148 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:37.331235 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":260,"threads_started":1}
I20260927 18:35:37.791116 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:37.790395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:54731)
0927 18:35:37.790857 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:37.790932 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260927 18:35:37.891239 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:37.890427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:49825)
0927 18:35:37.890966 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:37.891077 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":317,"threads_started":1}
I20260927 18:35:38.332868 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:38.332244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:49721)
0927 18:35:38.332634 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:38.332710 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260927 18:35:38.792912 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:38.792112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:39799)
0927 18:35:38.792589 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:38.792667 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":281,"threads_started":1}
I20260927 18:35:38.892725 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:38.892019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:45055)
0927 18:35:38.892484 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:38.892567 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260927 18:35:39.334411 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:39.333712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:56789)
0927 18:35:39.334151 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:39.334227 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260927 18:35:39.794579 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:39.793754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:60451)
0927 18:35:39.794311 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:39.794399 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":283,"threads_started":1}
I20260927 18:35:39.894532 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:39.893660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:53885)
0927 18:35:39.894243 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:39.894340 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":286,"threads_started":1}
I20260927 18:35:40.336112 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:40.335398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:51317)
0927 18:35:40.335866 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:40.335946 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260927 18:35:40.796154 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:40.795423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:56905)
0927 18:35:40.795897 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:40.795986 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260927 18:35:40.896185 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:40.895469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:44759)
0927 18:35:40.895928 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:40.896028 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260927 18:35:41.337615 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:41.336930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:53999)
0927 18:35:41.337370 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:41.337457 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260927 18:35:41.797942 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:41.797154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:36819)
0927 18:35:41.797624 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:41.797716 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260927 18:35:41.898111 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:41.897130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:49361)
0927 18:35:41.897731 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:41.897878 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":305,"threads_started":1}
I20260927 18:35:42.339510 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:42.338616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:59107)
0927 18:35:42.339219 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:42.339296 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":279,"threads_started":1}
I20260927 18:35:42.799654 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:42.798884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:36633)
0927 18:35:42.799369 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:42.799448 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260927 18:35:42.899623 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:42.898924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:52507)
0927 18:35:42.899386 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:42.899465 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260927 18:35:43.341121 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:43.340380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:44501)
0927 18:35:43.340835 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:43.340919 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260927 18:35:43.801234 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:43.800559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:42587)
0927 18:35:43.800991 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:43.801069 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260927 18:35:43.901268 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:43.900511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:46483)
0927 18:35:43.900992 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:43.901101 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260927 18:35:44.342779 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:44.342024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:51251)
0927 18:35:44.342507 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:44.342608 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260927 18:35:44.803117 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:44.802273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:51463)
0927 18:35:44.802846 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:44.802926 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":313,"threads_started":1}
I20260927 18:35:44.902838 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:44.902215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:60071)
0927 18:35:44.902605 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:44.902682 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":208,"threads_started":1}
I20260927 18:35:45.344504 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:45.343759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:53677)
0927 18:35:45.344243 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:45.344321 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":289,"threads_started":1}
I20260927 18:35:45.804767 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:45.804013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:38681)
0927 18:35:45.804507 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:45.804593 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":270,"threads_started":1}
I20260927 18:35:45.904567 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:45.903823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:35577)
0927 18:35:45.904280 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:45.904361 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260927 18:35:46.346390 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:46.345534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:38507)
0927 18:35:46.346095 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:46.346196 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":330,"threads_started":1}
I20260927 18:35:46.806805 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:46.806008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:43233)
0927 18:35:46.806511 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:46.806624 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260927 18:35:46.906530 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:46.905627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:60209)
0927 18:35:46.906264 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:46.906351 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":405,"threads_started":1}
I20260927 18:35:47.347929 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:47.347278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:39799)
0927 18:35:47.347687 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:47.347765 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":206,"threads_started":1}
I20260927 18:35:47.808360 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:47.807720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:35997)
0927 18:35:47.808131 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:47.808203 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260927 18:35:47.908176 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:47.907346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:39167)
0927 18:35:47.907848 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:47.907975 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":268,"threads_started":1}
I20260927 18:35:48.349700 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:48.348912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:37629)
0927 18:35:48.349407 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:48.349507 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":239,"threads_started":1}
I20260927 18:35:48.810120 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:48.809241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:43337)
0927 18:35:48.809744 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:48.809900 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260927 18:35:48.909842 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:48.909079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:33815)
0927 18:35:48.909524 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:48.909603 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":219,"threads_started":1}
I20260927 18:35:49.351274 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:49.350575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:50131)
0927 18:35:49.351025 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:49.351102 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260927 18:35:49.811694 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:49.810988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:41933)
0927 18:35:49.811422 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:49.811507 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260927 18:35:49.911679 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:49.910962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.65:43631)
0927 18:35:49.911401 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:49.911504 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260927 18:35:50.353139 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.352368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.66:57825)
0927 18:35:50.352798 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.352893 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
I20260927 18:35:50.790156 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15344
I20260927 18:35:50.814108 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.813063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37019 (local address 127.0.97.67:59467)
0927 18:35:50.813648 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.813837 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:37019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260927 18:35:50.815603 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.791450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.97.65:42777)
0927 18:35:50.792300 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.792323 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.792376 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:35:50.795459 (+ 3083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.795476 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.796142 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.796594 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:50.798147 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.800629 (+ 2482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.802109 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.803510 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.803730 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.806767 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:50.806781 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:50.806806 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:50.806862 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:50.811568 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.812309 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.812320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.812332 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.812582 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.813055 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.813066 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.813069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.813892 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:50.814255 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.814831 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.815278 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":467,"threads_started":1}
I20260927 18:35:50.817224 15502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:50.903515 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15344
I20260927 18:35:50.921361 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15510
I20260927 18:35:50.939185 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.922278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.97.66:41311)
0927 18:35:50.922809 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.922822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.922859 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:35:50.924335 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.924344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.924782 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.925076 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:50.926495 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.927795 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.929073 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.929765 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.929970 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.932821 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:50.932829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:50.932843 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:50.932879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:50.935991 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.936654 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.936660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.936666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.936820 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.937214 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.937217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.937219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.937739 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:50.938216 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.938574 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.938900 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":269,"threads_started":1}
I20260927 18:35:50.940557 15633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:51.024004 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15510
I20260927 18:35:51.041599 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15641
I20260927 18:35:51.059279 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:51.042624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.0.97.67:33341)
0927 18:35:51.043322 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:51.043337 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:51.043377 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:35:51.045001 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:51.045010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:51.045435 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:51.045735 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:51.047188 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:51.048435 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:51.049638 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:51.050386 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:51.050551 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:51.053047 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:51.053054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:51.053069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:51.053106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:51.056164 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:51.056754 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:51.056760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:51.056766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:51.056935 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:51.057263 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:51.057266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:51.057268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:51.057766 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:51.058093 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:51.058550 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 18:35:51.058973 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":373,"threads_started":1}
I20260927 18:35:51.060689 15764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:51.144129 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15641
2026-09-27T18:35:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24283 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:51 dist-test-slave-n3p4 krb5kdc[15826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:51 dist-test-slave-n3p4 krb5kdc[15826](info): set up 2 sockets
Sep 27 18:35:51 dist-test-slave-n3p4 krb5kdc[15826](info): commencing operation
krb5kdc: starting...
W20260927 18:35:52.250316 389 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:35:52 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:35:52.485730 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:52.816851 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:52.817129 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:52.817310 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:52.824468 15839 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:35:52.824565 15839 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:35:52.824628 15839 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:35:52.824702 15839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:52.824783 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:52.824898 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:52.824966 15839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:52.825050 15839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:52.834872 15839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:52.836205 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:52.837630 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:52.849854 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:52.849911 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:52.849877 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:52.851176 15839 server_base.cc:1034] running on GCE node
I20260927 18:35:52.852021 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:52.853533 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:52.854785 15839 hybrid_clock.cc:648] HybridClock initialized: now 1790534152854711 us; error 49 us; skew 500 ppm
I20260927 18:35:52.855177 15839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:52 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:52.858529 15839 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:35:52.860055 15839 webserver.cc:460] Webserver started at http://127.0.97.126:38047/ using document root <none> and password file <none>
I20260927 18:35:52.860754 15839 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:52.860846 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:52.861138 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:52.864513 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "5d9508c1d9214ae9bfa8f499aa7810a3"
format_stamp: "Formatted at 2026-09-27 18:35:52 on dist-test-slave-n3p4"
server_key: "4708aad9c3596e2172c466bb821719eb"
server_key_iv: "3d28e0b12590ea62ab79744e3c3873c5"
server_key_version: "encryptionkey@0"
I20260927 18:35:52.865376 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "5d9508c1d9214ae9bfa8f499aa7810a3"
format_stamp: "Formatted at 2026-09-27 18:35:52 on dist-test-slave-n3p4"
server_key: "4708aad9c3596e2172c466bb821719eb"
server_key_iv: "3d28e0b12590ea62ab79744e3c3873c5"
server_key_version: "encryptionkey@0"
I20260927 18:35:52.871052 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 18:35:52.874590 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:52.876099 15839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:35:52.876295 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5d9508c1d9214ae9bfa8f499aa7810a3"
format_stamp: "Formatted at 2026-09-27 18:35:52 on dist-test-slave-n3p4"
server_key: "4708aad9c3596e2172c466bb821719eb"
server_key_iv: "3d28e0b12590ea62ab79744e3c3873c5"
server_key_version: "encryptionkey@0"
I20260927 18:35:52.876518 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:52.904277 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:52.908732 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:52.908967 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:52.927762 15839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42279
I20260927 18:35:52.927735 15905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42279 every 8 connection(s)
I20260927 18:35:52.929159 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:52.933018 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:52.938216 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15839
I20260927 18:35:52.938429 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:35:52.939100 389 external_mini_cluster.cc:1428] Setting key 6d2280f3e973440b58ee4c91a83d33c1
I20260927 18:35:52.944921 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap starting.
I20260927 18:35:52.948696 15906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:52.949726 15906 log.cc:824] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:52.952528 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: No bootstrap required, opened a new log
Sep 27 18:35:52 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:52.961308 15906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:35:52.961872 15906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:52.962039 15906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Initialized, Role: FOLLOWER
I20260927 18:35:52.962740 15906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:35:52.963006 15906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:52.963147 15906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:52.963335 15906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:52.963657 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:52.941099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.0.97.126:42279)
0927 18:35:52.941976 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:52.941991 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:52.942044 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:35:52.944323 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:52.944351 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:52.944829 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:52.945273 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:52.946661 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:52.948720 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:52.950198 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:52.951050 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:52.951223 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:52.955773 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:52.955781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:52.955795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:52.955830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:52.959078 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:52.959908 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:52.959915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:52.959925 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:52.960138 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:52.960515 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:52.960518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:52.960520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:52.961077 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:52.961303 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:52.961892 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 18:35:52.962443 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":333,"threads_started":1}
I20260927 18:35:52.966223 15906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:35:52.966862 15906 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9508c1d9214ae9bfa8f499aa7810a3; no voters:
I20260927 18:35:52.967487 15906 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:52.967630 15911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:52.968379 15911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Running, Role: LEADER
I20260927 18:35:52.969004 15911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:35:52.969597 15906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:52.971990 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9508c1d9214ae9bfa8f499aa7810a3. Latest consensus state: current_term: 1 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:35:52.972033 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:35:52.972443 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:52.972440 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:52.983434 15918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:52.989544 15918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:52.998611 15918 catalog_manager.cc:1347] Generated new cluster ID: 30e543dcc3204265aac8686c757a5d7d
I20260927 18:35:52.998730 15918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:53.030215 15918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:53.032146 15918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:53.039503 15918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Generated new TSK 0
I20260927 18:35:53.040333 15918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:35:53.101733 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:53.448887 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:53.449182 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:53.449282 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:53.456672 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:53.456786 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:53.456956 15932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:35:53.467617 15932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:53.469012 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:53.470440 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:53.485109 15938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:53.485087 15940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:53.485411 15937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:53.486361 15932 server_base.cc:1034] running on GCE node
I20260927 18:35:53.486965 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:53.487912 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:53.489158 15932 hybrid_clock.cc:648] HybridClock initialized: now 1790534153489081 us; error 66 us; skew 500 ppm
I20260927 18:35:53.489490 15932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:53 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:53.492941 15932 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:35:53.494545 15932 webserver.cc:460] Webserver started at http://127.0.97.65:45691/ using document root <none> and password file <none>
I20260927 18:35:53.495254 15932 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:53.495395 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:53.495707 15932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:53.498889 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
format_stamp: "Formatted at 2026-09-27 18:35:53 on dist-test-slave-n3p4"
server_key: "f78837b51941dc20b9b41c1f800556d6"
server_key_iv: "066081d0ae2ad951ba7c5856359a31f4"
server_key_version: "encryptionkey@0"
I20260927 18:35:53.499683 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
format_stamp: "Formatted at 2026-09-27 18:35:53 on dist-test-slave-n3p4"
server_key: "f78837b51941dc20b9b41c1f800556d6"
server_key_iv: "066081d0ae2ad951ba7c5856359a31f4"
server_key_version: "encryptionkey@0"
I20260927 18:35:53.504913 15932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:35:53.508469 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:53.509748 15932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:35:53.509974 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
format_stamp: "Formatted at 2026-09-27 18:35:53 on dist-test-slave-n3p4"
server_key: "f78837b51941dc20b9b41c1f800556d6"
server_key_iv: "066081d0ae2ad951ba7c5856359a31f4"
server_key_version: "encryptionkey@0"
I20260927 18:35:53.510185 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:53.525827 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:53.530685 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:53.530989 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:53.531951 15932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:53.533330 15932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:53.533403 15932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:53.533560 15932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:53.533628 15932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:53.567718 15932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33459
I20260927 18:35:53.567823 16059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33459 every 8 connection(s)
I20260927 18:35:53.569140 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:53.579038 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 15932
I20260927 18:35:53.579257 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:53.579937 389 external_mini_cluster.cc:1428] Setting key dda21d9f336bf60a939e3635aa2f7cfc
Sep 27 18:35:53 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:53.590404 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:53.571972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49307 (local address 127.0.97.126:42279)
0927 18:35:53.572377 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:53.572382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:53.573447 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 18:35:53.575569 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:53.575578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:53.575734 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:53.575880 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:53.577434 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:53.578230 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:53.579895 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:53.580340 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:53.580466 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:53.584607 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:53.584615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:53.584623 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:53.584655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:53.587329 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:53.588075 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:53.588080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:53.588082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:53.588193 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:53.588618 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:53.588624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:53.588628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:53.588893 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:53.589055 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:53.589755 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 18:35:53.590030 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":196,"threads_started":1}
I20260927 18:35:53.591594 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:53.572586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:49307)
0927 18:35:53.573289 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:53.573330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:53.575068 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:53.576152 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:53.576161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:53.576716 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:53.577269 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:53.577286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:53.578402 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:53.578406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:53.579748 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:53.579759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:53.580631 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:53.580641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:53.580857 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:53.581595 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:53.581618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:53.584354 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:53.587476 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:53.587486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:53.587512 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:53.587934 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:53.588300 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:53.588306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:53.588310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:53.588509 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:53.589020 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:53.589028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:53.589380 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 18:35:53.589647 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 18:35:53.590212 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":270,"threads_started":1}
I20260927 18:35:53.593318 16060 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:35:53.593662 16060 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:53.594583 16060 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:35:53.597095 15871 ts_manager.cc:194] Registered new tserver with Master: 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459)
I20260927 18:35:53.599349 15871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49307
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:35:53.633365 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:53.980280 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:53.980652 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:53.980765 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:53.988235 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:53.988348 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:53.988467 16066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:35:53.998023 16066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:53.999305 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:54.000581 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:54.013921 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:54.013970 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:54.013974 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:54.014823 16066 server_base.cc:1034] running on GCE node
I20260927 18:35:54.015376 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:54.016256 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:54.017443 16066 hybrid_clock.cc:648] HybridClock initialized: now 1790534154017376 us; error 57 us; skew 500 ppm
I20260927 18:35:54.017750 16066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:54 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:54.020812 16066 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:35:54.022296 16066 webserver.cc:460] Webserver started at http://127.0.97.66:38983/ using document root <none> and password file <none>
I20260927 18:35:54.022960 16066 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:54.023104 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:54.023420 16066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:54.026304 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "57f24691d17f4a3d82da9053fb274390"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "3d4725454ca0c6c18d35d0e858575d3a"
server_key_iv: "d7e41257819463f98e683c3b8291e891"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.027050 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "57f24691d17f4a3d82da9053fb274390"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "3d4725454ca0c6c18d35d0e858575d3a"
server_key_iv: "d7e41257819463f98e683c3b8291e891"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.032161 16066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:35:54.035734 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.037338 16066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:35:54.037546 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "57f24691d17f4a3d82da9053fb274390"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "3d4725454ca0c6c18d35d0e858575d3a"
server_key_iv: "d7e41257819463f98e683c3b8291e891"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.037772 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:54.058586 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:54.063194 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:54.063521 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:54.064555 16066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:54.066020 16066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:54.066118 16066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.066262 16066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:54.066330 16066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.100937 16066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39623
I20260927 18:35:54.101006 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39623 every 8 connection(s)
I20260927 18:35:54.102264 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:54.107611 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16066
I20260927 18:35:54.107801 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:54.108428 389 external_mini_cluster.cc:1428] Setting key 176d0f6f668aeceba71ffac2727d7710
Sep 27 18:35:54 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:54.123320 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.104650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36401 (local address 127.0.97.126:42279)
0927 18:35:54.105081 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.105085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.106113 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 18:35:54.107993 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.107999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.108086 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.108211 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:54.109700 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.110517 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.112013 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.112430 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.112569 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.116307 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:54.116318 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:54.116321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:54.116353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:54.120015 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.120731 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.120739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.120745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.120911 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.121303 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.121311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.121318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.121723 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:54.121934 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.122550 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.122835 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260927 18:35:54.123905 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.105200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:36401)
0927 18:35:54.105947 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.105993 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.107663 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.108391 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.108399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.108964 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:54.109530 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.109547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.110659 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.110669 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.111860 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.111871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.112655 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.112664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.112889 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.113562 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:54.113588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:54.116087 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:54.120164 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:54.120171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:54.120183 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:54.120613 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:54.121021 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:54.121024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:54.121027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:54.121199 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:54.121878 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:54.121884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:54.122188 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.122417 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.122803 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":260,"threads_started":1}
I20260927 18:35:54.125342 16194 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:35:54.125644 16194 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:54.126369 16194 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:35:54.127799 15871 ts_manager.cc:194] Registered new tserver with Master: 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623)
I20260927 18:35:54.128736 15871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:36401
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:35:54.163162 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:54.487805 16200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:54.488140 16200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:54.488300 16200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:54.495743 16200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:54.495913 16200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:54.496095 16200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:35:54.506263 16200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:35:54.507652 16200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:54.508991 16200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:54.521677 16205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:54.521759 16206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:54.521894 16208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:54.522495 16200 server_base.cc:1034] running on GCE node
I20260927 18:35:54.523121 16200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:54.524036 16200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:54.525250 16200 hybrid_clock.cc:648] HybridClock initialized: now 1790534154525194 us; error 52 us; skew 500 ppm
I20260927 18:35:54.525550 16200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:35:54 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:54.534192 16200 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:35:54.535738 16200 webserver.cc:460] Webserver started at http://127.0.97.67:35943/ using document root <none> and password file <none>
I20260927 18:35:54.536401 16200 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:54.536548 16200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:54.536863 16200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:54.539865 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "628aa59f1faa4072bb345be3dad4b782"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "18c55d44a815563ead76eb05e06efc1e"
server_key_iv: "78ab8dce5365ee2ce1cad58532fbb8c4"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.540617 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "628aa59f1faa4072bb345be3dad4b782"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "18c55d44a815563ead76eb05e06efc1e"
server_key_iv: "78ab8dce5365ee2ce1cad58532fbb8c4"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.545770 16200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:35:54.549122 16215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.550498 16200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:54.550658 16200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "628aa59f1faa4072bb345be3dad4b782"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "18c55d44a815563ead76eb05e06efc1e"
server_key_iv: "78ab8dce5365ee2ce1cad58532fbb8c4"
server_key_version: "encryptionkey@0"
I20260927 18:35:54.550846 16200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:54.576395 16200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:54.580369 16200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:54.580657 16200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:54.581661 16200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:54.583029 16200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:54.583133 16200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.583274 16200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:54.583343 16200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:54.602671 16060 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:35:54.618113 16200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44951
I20260927 18:35:54.618158 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44951 every 8 connection(s)
I20260927 18:35:54.619508 16200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:54.626271 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16200
I20260927 18:35:54.626458 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:54.626865 389 external_mini_cluster.cc:1428] Setting key 32ef776e823f7c14875cc12fca44d634
Sep 27 18:35:54 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:35:54.638813 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.622199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51247 (local address 127.0.97.126:42279)
0927 18:35:54.622382 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.622386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.623525 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 18:35:54.625374 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.625379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.625481 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.625596 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:54.627222 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.628101 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.629601 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.630046 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.630176 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.633495 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:54.633500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:54.633503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:54.633534 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:54.635950 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.636620 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.636626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.636630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.636744 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.637146 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.637151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.637155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.637425 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:54.637593 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.638265 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.638482 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:35:54.639708 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.622654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:51247)
0927 18:35:54.623369 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.623412 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.625111 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.625880 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.625888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.626431 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:54.627058 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.627083 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.628231 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.628237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.629449 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.629460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.630254 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.630261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.630482 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.631070 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:54.631141 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:54.633301 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:54.636086 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:54.636094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:54.636110 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:54.636502 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:54.636846 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:54.636852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:54.636855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:54.637026 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:54.637553 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:54.637561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:54.637919 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.638177 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.638676 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":295,"threads_started":1}
I20260927 18:35:54.641151 16328 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:35:54.641510 16328 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:54.642390 16328 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:35:54.643901 15871 ts_manager.cc:194] Registered new tserver with Master: 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951)
I20260927 18:35:54.644701 15871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51247
I20260927 18:35:54.654178 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:54.671833 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.658722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.97.126:42279)
0927 18:35:54.659056 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.659065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.659479 (+ 414us) server_negotiation.cc:407] Connection header received
0927 18:35:54.659661 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.659666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.659751 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.659852 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:54.661339 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.662197 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.663548 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.663981 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.664115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.666630 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:54.666636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:54.666641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:54.666681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:54.669092 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.669733 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.669738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.669742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.669914 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.670266 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.670274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.670279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.670602 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:54.670860 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.671192 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.671422 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:35:54.675813 15871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:54.682849 15871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:54.706179 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.695263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.0.97.65:33459)
0927 18:35:54.698224 (+ 2961us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.698235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.698268 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:35:54.698380 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.698390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.698672 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.700274 (+ 1602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:54.700526 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.701628 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.703860 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.704982 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.705134 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.705384 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.705636 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.705837 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2738,"thread_start_us":2508,"threads_started":1}
I20260927 18:35:54.706292 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.694961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33459 (local address 127.0.0.1:56290)
0927 18:35:54.695833 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.695898 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.696160 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.698931 (+ 2771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.698938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.698977 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:54.699595 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.699610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.701769 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.701774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.703666 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.703683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.705194 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.705200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.705389 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.705450 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.705584 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.705839 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":80,"thread_start_us":1520,"threads_started":1}
I20260927 18:35:54.706826 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.695100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39623 (local address 127.0.0.1:59606)
0927 18:35:54.696803 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.696831 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.696953 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.698222 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.698234 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.698278 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:54.699081 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.699124 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.701526 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.701531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.704110 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.704125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.706003 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.706013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.706209 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.706246 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.706313 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.706433 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":115,"thread_start_us":256,"threads_started":1}
I20260927 18:35:54.707875 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.695416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.97.66:39623)
0927 18:35:54.696971 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.696980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.697010 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:35:54.697120 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.697130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.697394 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.697584 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:54.699797 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.701378 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.704301 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.705740 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.707148 (+ 1408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.707332 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.707397 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.707546 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":1994,"threads_started":1}
I20260927 18:35:54.710641 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.696930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44951 (local address 127.0.0.1:52966)
0927 18:35:54.700286 (+ 3356us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.700308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.700494 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.701316 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.701322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.701347 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:54.701761 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.701812 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.703755 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.703760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.705936 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.705949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.709962 (+ 4013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.709966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.710091 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.710111 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.710200 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.710322 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3240,"thread_start_us":3051,"threads_started":1}
I20260927 18:35:54.710765 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.697550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.0.97.67:44951)
0927 18:35:54.699809 (+ 2259us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.699816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.700400 (+ 584us) server_negotiation.cc:407] Connection header received
0927 18:35:54.700658 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.700665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.700875 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.701014 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:54.701999 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.703594 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.706115 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.707522 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.707623 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.707842 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.710257 (+ 2415us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.710417 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2102,"thread_start_us":1955,"threads_started":1}
I20260927 18:35:54.710630 15995 tablet_service.cc:1467] Processing CreateTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf (DEFAULT_TABLE table=test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:54.711206 16129 tablet_service.cc:1467] Processing CreateTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf (DEFAULT_TABLE table=test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:54.712169 15995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91c1908f1a945e4a1b48fe8d34c69bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:54.712574 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91c1908f1a945e4a1b48fe8d34c69bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:54.713990 16263 tablet_service.cc:1467] Processing CreateTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf (DEFAULT_TABLE table=test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:54.715216 16263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91c1908f1a945e4a1b48fe8d34c69bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:54.726925 16346 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Bootstrap starting.
I20260927 18:35:54.729146 16348 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Bootstrap starting.
I20260927 18:35:54.730537 16346 tablet_bootstrap.cc:654] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:54.731838 16346 log.cc:824] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:54.732738 16348 tablet_bootstrap.cc:654] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:54.733237 16347 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Bootstrap starting.
I20260927 18:35:54.734010 16348 log.cc:824] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:54.735201 16346 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: No bootstrap required, opened a new log
I20260927 18:35:54.735482 16346 ts_tablet_manager.cc:1397] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:35:54.736964 16348 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: No bootstrap required, opened a new log
I20260927 18:35:54.737214 16347 tablet_bootstrap.cc:654] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:54.737231 16348 ts_tablet_manager.cc:1397] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 18:35:54.738180 16347 log.cc:824] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:54.740450 16347 tablet_bootstrap.cc:492] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: No bootstrap required, opened a new log
I20260927 18:35:54.740783 16347 ts_tablet_manager.cc:1397] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 18:35:54.746603 16346 raft_consensus.cc:348] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.747081 16346 raft_consensus.cc:374] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:54.747223 16346 raft_consensus.cc:729] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628aa59f1faa4072bb345be3dad4b782, State: Initialized, Role: FOLLOWER
I20260927 18:35:54.747870 16346 consensus_queue.cc:260] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.747917 16348 raft_consensus.cc:348] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.748332 16348 raft_consensus.cc:374] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:54.748514 16348 raft_consensus.cc:729] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f24691d17f4a3d82da9053fb274390, State: Initialized, Role: FOLLOWER
I20260927 18:35:54.748994 16328 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:35:54.749224 16348 consensus_queue.cc:260] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.749179 16347 raft_consensus.cc:348] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.749569 16347 raft_consensus.cc:374] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:54.749652 16347 raft_consensus.cc:729] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec6d338f884b1aa1e9fbca1c95477e, State: Initialized, Role: FOLLOWER
I20260927 18:35:54.749827 16346 ts_tablet_manager.cc:1428] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260927 18:35:54.750193 16347 consensus_queue.cc:260] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.750855 16194 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:35:54.751495 16348 ts_tablet_manager.cc:1428] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 18:35:54.754179 16347 ts_tablet_manager.cc:1428] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260927 18:35:54.821688 16061 tablet.cc:2369] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:54.851696 16353 raft_consensus.cc:482] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:54.852020 16353 raft_consensus.cc:504] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.853752 16353 leader_election.cc:288] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459), 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951)
W20260927 18:35:54.854341 16195 tablet.cc:2369] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:54.860562 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.854473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40939 (local address 127.0.97.65:33459)
0927 18:35:54.854709 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.854725 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.855147 (+ 422us) server_negotiation.cc:407] Connection header received
0927 18:35:54.855482 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.855490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.855571 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.855640 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:54.856293 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.857086 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.858881 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.859658 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.859813 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.859927 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.860181 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.860305 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:35:54.860702 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.854149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33459 (local address 127.0.97.66:40939)
0927 18:35:54.855027 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.855062 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.855279 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.855850 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.855855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.855886 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:54.856157 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.856164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.857204 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.857212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.858732 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.858743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.859882 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.859888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.860026 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.860055 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.860156 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.860277 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":3,"thread_start_us":601,"threads_started":1}
I20260927 18:35:54.861449 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.855625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44427 (local address 127.0.97.67:44951)
0927 18:35:54.855843 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.855849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.855871 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:35:54.855944 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.855948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.856055 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.856169 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:54.856759 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.857673 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.859525 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.860360 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.860524 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.860635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.861089 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.861180 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:35:54.861528 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.854256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44951 (local address 127.0.97.66:44427)
0927 18:35:54.855564 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:54.855579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:54.855720 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:54.856194 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:54.856199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:54.856224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:54.856612 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.856622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.857840 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.857845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.859363 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:54.859375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.860593 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.860600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:54.860896 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.860957 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:35:54.861069 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:35:54.861199 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":19,"thread_start_us":946,"threads_started":1}
I20260927 18:35:54.861761 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91c1908f1a945e4a1b48fe8d34c69bf" candidate_uuid: "57f24691d17f4a3d82da9053fb274390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" is_pre_election: true
I20260927 18:35:54.862386 16015 raft_consensus.cc:2454] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f24691d17f4a3d82da9053fb274390 in term 0.
I20260927 18:35:54.862565 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91c1908f1a945e4a1b48fe8d34c69bf" candidate_uuid: "57f24691d17f4a3d82da9053fb274390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aa59f1faa4072bb345be3dad4b782" is_pre_election: true
I20260927 18:35:54.863067 16283 raft_consensus.cc:2454] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f24691d17f4a3d82da9053fb274390 in term 0.
I20260927 18:35:54.863071 16082 leader_election.cc:302] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f24691d17f4a3d82da9053fb274390, 77ec6d338f884b1aa1e9fbca1c95477e; no voters:
I20260927 18:35:54.863420 16353 raft_consensus.cc:2781] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:54.863564 16353 raft_consensus.cc:482] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:54.863698 16353 raft_consensus.cc:3037] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:54.865855 16353 raft_consensus.cc:504] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.866480 16353 leader_election.cc:288] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [CANDIDATE]: Term 1 election: Requested vote from peers 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459), 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951)
I20260927 18:35:54.866825 16015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91c1908f1a945e4a1b48fe8d34c69bf" candidate_uuid: "57f24691d17f4a3d82da9053fb274390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
I20260927 18:35:54.866973 16283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91c1908f1a945e4a1b48fe8d34c69bf" candidate_uuid: "57f24691d17f4a3d82da9053fb274390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aa59f1faa4072bb345be3dad4b782"
I20260927 18:35:54.867025 16015 raft_consensus.cc:3037] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:54.867172 16283 raft_consensus.cc:3037] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:54.870177 16015 raft_consensus.cc:2454] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f24691d17f4a3d82da9053fb274390 in term 1.
I20260927 18:35:54.870232 16283 raft_consensus.cc:2454] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f24691d17f4a3d82da9053fb274390 in term 1.
I20260927 18:35:54.870572 16082 leader_election.cc:302] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f24691d17f4a3d82da9053fb274390, 77ec6d338f884b1aa1e9fbca1c95477e; no voters:
I20260927 18:35:54.870868 16353 raft_consensus.cc:2781] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 18:35:54.871181 16329 tablet.cc:2369] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:35:54.871351 16353 raft_consensus.cc:686] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 1 LEADER]: Becoming Leader. State: Replica: 57f24691d17f4a3d82da9053fb274390, State: Running, Role: LEADER
I20260927 18:35:54.871865 16353 consensus_queue.cc:237] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } }
I20260927 18:35:54.876413 15869 catalog_manager.cc:5495] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57f24691d17f4a3d82da9053fb274390 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "57f24691d17f4a3d82da9053fb274390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } health_report { overall_health: UNKNOWN } } }
I20260927 18:35:54.897178 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.891994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.0.97.66:39623)
0927 18:35:54.892193 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.892198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.892215 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:35:54.892387 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.892391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.892474 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.892611 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:54.892933 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.893652 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.895124 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.895547 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.895660 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.896126 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:54.896616 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:54.896725 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.896819 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.896882 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:35:54.905294 16283 raft_consensus.cc:1261] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Refusing update from remote peer 57f24691d17f4a3d82da9053fb274390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:54.905406 16015 raft_consensus.cc:1261] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Refusing update from remote peer 57f24691d17f4a3d82da9053fb274390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:54.906380 16353 consensus_queue.cc:1035] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:54.906661 16357 consensus_queue.cc:1035] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:54.918785 16360 mvcc.cc:205] Tried to move back new op lower bound from 7334027898482831360 to 7334027898358861824. Current Snapshot: MvccSnapshot[applied={T|T < 7334027898482831360}]
I20260927 18:35:54.935681 15871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:54.936018 15871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:54.940301 15871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Sending DeleteTablet for 3 replicas of tablet a91c1908f1a945e4a1b48fe8d34c69bf
I20260927 18:35:54.941620 16129 tablet_service.cc:1514] Processing DeleteTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:59606
I20260927 18:35:54.941498 15995 tablet_service.cc:1514] Processing DeleteTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56290
I20260927 18:35:54.944159 16263 tablet_service.cc:1514] Processing DeleteTablet for tablet a91c1908f1a945e4a1b48fe8d34c69bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52966
I20260927 18:35:54.951731 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15932
I20260927 18:35:54.951068 16370 tablet_replica.cc:330] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: stopping tablet replica
I20260927 18:35:54.953881 16370 raft_consensus.cc:2229] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:35:54.955464 16370 raft_consensus.cc:2258] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:54.956453 16372 tablet_replica.cc:330] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: stopping tablet replica
I20260927 18:35:54.958330 16372 raft_consensus.cc:2229] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:54.959385 16372 raft_consensus.cc:2258] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:54.959631 16370 ts_tablet_manager.cc:1905] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:54.962291 16372 ts_tablet_manager.cc:1905] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:54.968566 16370 ts_tablet_manager.cc:1918] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:54.968791 16370 log.cc:1196] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/a91c1908f1a945e4a1b48fe8d34c69bf
I20260927 18:35:54.969516 16370 ts_tablet_manager.cc:1939] T a91c1908f1a945e4a1b48fe8d34c69bf P 57f24691d17f4a3d82da9053fb274390: Deleting consensus metadata
I20260927 18:35:54.970337 16371 tablet_replica.cc:330] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: stopping tablet replica
I20260927 18:35:54.970934 16371 raft_consensus.cc:2229] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:54.971643 16371 raft_consensus.cc:2258] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:54.971607 15856 catalog_manager.cc:4841] TS 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623): tablet a91c1908f1a945e4a1b48fe8d34c69bf (table test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]) successfully deleted
I20260927 18:35:54.973764 16372 ts_tablet_manager.cc:1918] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:54.973995 16372 log.cc:1196] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/a91c1908f1a945e4a1b48fe8d34c69bf
I20260927 18:35:54.974557 16372 ts_tablet_manager.cc:1939] T a91c1908f1a945e4a1b48fe8d34c69bf P 628aa59f1faa4072bb345be3dad4b782: Deleting consensus metadata
I20260927 18:35:54.974738 16371 ts_tablet_manager.cc:1905] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:54.976296 15858 catalog_manager.cc:4841] TS 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951): tablet a91c1908f1a945e4a1b48fe8d34c69bf (table test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]) successfully deleted
Sep 27 18:35:54 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:35:54.982622 16371 ts_tablet_manager.cc:1918] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:54.982832 16371 log.cc:1196] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/a91c1908f1a945e4a1b48fe8d34c69bf
I20260927 18:35:54.983479 16371 ts_tablet_manager.cc:1939] T a91c1908f1a945e4a1b48fe8d34c69bf P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting consensus metadata
I20260927 18:35:54.985725 15855 catalog_manager.cc:4841] TS 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459): tablet a91c1908f1a945e4a1b48fe8d34c69bf (table test-table [id=8b9d4b86a1f24fefa932ce62b6aa1284]) successfully deleted
I20260927 18:35:54.987691 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:54.952919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.0.97.65:33459)
0927 18:35:54.953166 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:54.953175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:54.968481 (+ 15306us) server_negotiation.cc:407] Connection header received
0927 18:35:54.969097 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:54.969104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:54.969201 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:54.969272 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:54.971494 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.972702 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.975238 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:54.975884 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:54.976004 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:54.980906 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:54.980917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:54.980935 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:54.980974 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:54.984432 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.985354 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.985361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.985363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.985553 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:54.986098 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:54.986103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:54.986107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:54.986451 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:54.986762 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:54.987090 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 18:35:54.987305 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:35:54.988325 16055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:55.103086 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15932
I20260927 18:35:55.127439 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16066
Sep 27 18:35:55 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:35:55.146402 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:55.128244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.0.97.66:39623)
0927 18:35:55.128415 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:55.128419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:55.128528 (+ 109us) server_negotiation.cc:407] Connection header received
0927 18:35:55.128733 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:55.128736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:55.128796 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:55.128876 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:55.130958 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:55.131803 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:55.134020 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:55.134511 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:55.134606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:55.139854 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:55.139865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:55.139879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:55.139914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:55.143536 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:55.144237 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:55.144246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:55.144250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:55.144417 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:55.144838 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:55.144842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:55.144846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:55.145195 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:55.145544 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:55.145829 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:35:55.146043 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:35:55.147010 16189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:55.262465 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16066
I20260927 18:35:55.279027 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16200
Sep 27 18:35:55 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:35:55.294569 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:55.279812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.0.97.67:44951)
0927 18:35:55.280010 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:55.280018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:55.280107 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:35:55.280324 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:55.280328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:55.280412 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:55.280530 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:55.281811 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:55.282521 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:55.284035 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:55.284470 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:55.284597 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:55.289208 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:55.289223 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:55.289239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:55.289278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:55.292114 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:55.292786 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:55.292793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:55.292796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:55.292945 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:55.293278 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:55.293281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:55.293283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:55.293622 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:55.293909 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:55.294150 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:35:55.294306 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:35:55.295171 16323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:55.387780 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16200
I20260927 18:35:55.406029 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 15839
I20260927 18:35:55.407199 15901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:35:55.519493 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 15839
I20260927 18:36:15.541383 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--webserver_interface=127.0.97.126
--webserver_port=38047
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:15.868443 16378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:15.868724 16378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:15.868889 16378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:15.876020 16378 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:36:15.876114 16378 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:36:15.876188 16378 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:36:15.876268 16378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:15.876349 16378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:15.876439 16378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:15.876498 16378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:15.876607 16378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:15.886860 16378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:15.888164 16378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:15.889505 16378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:15.912698 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:15.912767 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:15.912766 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:15.913084 16378 server_base.cc:1034] running on GCE node
I20260927 18:36:15.914314 16378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:15.915938 16378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:15.917223 16378 hybrid_clock.cc:648] HybridClock initialized: now 1790534175917146 us; error 46 us; skew 500 ppm
I20260927 18:36:15.917652 16378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:15 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:15.925202 16378 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:36:15.926964 16378 webserver.cc:460] Webserver started at http://127.0.97.126:38047/ using document root <none> and password file <none>
I20260927 18:36:15.927723 16378 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:15.927824 16378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:15.933686 16378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:36:15.936653 16394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:15.938179 16378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:15.938364 16378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5d9508c1d9214ae9bfa8f499aa7810a3"
format_stamp: "Formatted at 2026-09-27 18:35:52 on dist-test-slave-n3p4"
server_key: "4708aad9c3596e2172c466bb821719eb"
server_key_iv: "3d28e0b12590ea62ab79744e3c3873c5"
server_key_version: "encryptionkey@0"
I20260927 18:36:15.939435 16378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:15.954317 16378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:15.958931 16378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:15.959261 16378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:15.979233 16378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42279
I20260927 18:36:15.979256 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42279 every 8 connection(s)
I20260927 18:36:15.980970 16378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:15.984586 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16378
I20260927 18:36:15.985491 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:33459
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=45691
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:15.989331 16446 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:36:15.994565 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap starting.
I20260927 18:36:16.014413 16446 log.cc:824] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:16.026356 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:16.026911 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap complete.
I20260927 18:36:16.036302 16446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:16.037170 16446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Initialized, Role: FOLLOWER
I20260927 18:36:16.037751 16446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:16.038065 16446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:16.038196 16446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:16.038386 16446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:36:16.042492 16446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:16.043174 16446 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9508c1d9214ae9bfa8f499aa7810a3; no voters:
I20260927 18:36:16.044011 16446 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:36:16.044201 16450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:36:16.045063 16450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 LEADER]: Becoming Leader. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Running, Role: LEADER
I20260927 18:36:16.045827 16450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:16.050266 16451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9508c1d9214ae9bfa8f499aa7810a3. Latest consensus state: current_term: 2 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:36:16.050637 16451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:16.050968 16452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:36:16.051322 16452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:16.051203 16446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:16.066164 16454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:16.075024 16454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:16.076525 16454 catalog_manager.cc:1259] Loaded cluster ID: 30e543dcc3204265aac8686c757a5d7d
I20260927 18:36:16.076735 16454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:16.080807 16454 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:16.082392 16454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Loaded TSK: 0
I20260927 18:36:16.113966 16454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Generated new TSK 1
I20260927 18:36:16.114377 16454 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:36:16.343376 16448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:16.343711 16448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:16.343901 16448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:16.351506 16448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:16.351617 16448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:16.351830 16448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:36:16.362329 16448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:33459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=45691
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:16.363626 16448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:16.364984 16448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:16.380174 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:16.380599 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:16.380628 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:16.381438 16448 server_base.cc:1034] running on GCE node
I20260927 18:36:16.382200 16448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:16.383215 16448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:16.384469 16448 hybrid_clock.cc:648] HybridClock initialized: now 1790534176384390 us; error 65 us; skew 500 ppm
I20260927 18:36:16.384816 16448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:16 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:16.388281 16448 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:36:16.389832 16448 webserver.cc:460] Webserver started at http://127.0.97.65:45691/ using document root <none> and password file <none>
I20260927 18:36:16.390561 16448 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:16.390704 16448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:16.396178 16448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:16.399068 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.400513 16448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:36:16.400709 16448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
format_stamp: "Formatted at 2026-09-27 18:35:53 on dist-test-slave-n3p4"
server_key: "f78837b51941dc20b9b41c1f800556d6"
server_key_iv: "066081d0ae2ad951ba7c5856359a31f4"
server_key_version: "encryptionkey@0"
I20260927 18:36:16.401731 16448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:16.422464 16448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:16.426754 16448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:16.427067 16448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:16.428045 16448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:16.429337 16448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:16.429456 16448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.429587 16448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:16.429674 16448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.463531 16448 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33459
I20260927 18:36:16.463617 16595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33459 every 8 connection(s)
I20260927 18:36:16.464944 16448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:16.473596 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16448
I20260927 18:36:16.474412 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:39623
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=38983
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:16 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:16.489761 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:16.468003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:53867)
0927 18:36:16.468749 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:16.468791 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:16.470512 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:16.471777 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:16.471784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:16.472341 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:16.472856 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:16.472873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.475412 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.475416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:16.476628 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:16.476640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.478148 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.478154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:16.478344 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:16.479041 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:16.479069 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:16.481160 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:16.485879 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:16.485885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:16.485896 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:16.486319 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:16.486805 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:16.486808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:16.486811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:16.487003 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:16.487914 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:16.487920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:16.488223 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 18:36:16.488444 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 18:36:16.488808 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":305,"threads_started":1}
I20260927 18:36:16.490370 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:16.467865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53867 (local address 127.0.97.126:42279)
0927 18:36:16.468483 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:16.468500 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:16.468964 (+ 464us) server_negotiation.cc:407] Connection header received
0927 18:36:16.471013 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:16.471026 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:16.471342 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:16.471637 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:16.473079 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.475222 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.476777 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.477922 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.478123 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:16.481481 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:16.481492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:16.481512 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:16.481565 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:16.485708 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:16.486462 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:16.486473 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:16.486483 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:16.486689 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:16.487109 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:16.487117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:16.487123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:16.487785 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:16.487967 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:16.488704 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 18:36:16.489147 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":226,"threads_started":1}
I20260927 18:36:16.493216 16596 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:36:16.493541 16596 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:16.494302 16596 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:36:16.496476 16411 ts_manager.cc:194] Registered new tserver with Master: 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459)
I20260927 18:36:16.498025 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:53867
W20260927 18:36:16.809772 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:16.810052 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:16.810221 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:16.817165 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:16.817298 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:16.817481 16600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:36:16.826987 16600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:39623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=38983
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:16.828344 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:16.829794 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:16.842748 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:16.842795 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:16.842824 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:16.843703 16600 server_base.cc:1034] running on GCE node
I20260927 18:36:16.844342 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:16.845283 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:16.846509 16600 hybrid_clock.cc:648] HybridClock initialized: now 1790534176846453 us; error 48 us; skew 500 ppm
I20260927 18:36:16.846856 16600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:16 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:16.850442 16600 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:36:16.852036 16600 webserver.cc:460] Webserver started at http://127.0.97.66:38983/ using document root <none> and password file <none>
I20260927 18:36:16.852764 16600 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:16.852881 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:16.858358 16600 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:36:16.861027 16615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.862509 16600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:16.862648 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "57f24691d17f4a3d82da9053fb274390"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "3d4725454ca0c6c18d35d0e858575d3a"
server_key_iv: "d7e41257819463f98e683c3b8291e891"
server_key_version: "encryptionkey@0"
I20260927 18:36:16.863636 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:16.891059 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:16.895556 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:16.895789 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:16.896778 16600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:16.898234 16600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:16.898358 16600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.898473 16600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:16.898533 16600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:16.932010 16600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39623
I20260927 18:36:16.932087 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39623 every 8 connection(s)
I20260927 18:36:16.933446 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:16.941591 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16600
I20260927 18:36:16.942377 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:44951
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=35943
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:16 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:16.955088 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:16.936169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57373 (local address 127.0.97.126:42279)
0927 18:36:16.936347 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:16.936352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:16.937492 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 18:36:16.939321 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:16.939325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:16.939392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:16.939488 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:16.940913 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.942024 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.943773 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.944544 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.944700 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:16.948460 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:16.948469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:16.948484 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:16.948536 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:16.951484 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:16.952539 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:16.952544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:16.952546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:16.952673 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:16.953172 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:16.953175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:16.953177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:16.953459 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:16.953580 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:16.954554 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 18:36:16.954764 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:36:16.956597 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:16.936659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:57373)
0927 18:36:16.937332 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:16.937373 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:16.939062 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:16.939695 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:16.939703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:16.940206 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:16.940757 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:16.940773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.942171 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.942175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:16.943639 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:16.943650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:16.944745 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:16.944751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:16.944939 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:16.945652 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:16.945674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:16.948195 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:16.951629 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:16.951642 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:16.951663 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:16.952357 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:16.952792 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:16.952799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:16.952805 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:16.953057 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:16.953610 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:16.953621 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:16.954108 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 18:36:16.954441 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 18:36:16.955051 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":261,"threads_started":1}
I20260927 18:36:16.958159 16728 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:36:16.958503 16728 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:16.959228 16728 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:36:16.960675 16411 ts_manager.cc:194] Registered new tserver with Master: 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623)
I20260927 18:36:16.961481 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:57373
W20260927 18:36:17.278276 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:17.278587 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:17.278745 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:17.286314 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:17.286441 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:17.286640 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:36:17.296550 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:44951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=35943
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:17.297884 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:17.299201 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:17.312453 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:17.312497 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:17.312639 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:17.313108 16731 server_base.cc:1034] running on GCE node
I20260927 18:36:17.313728 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:17.314725 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:17.315999 16731 hybrid_clock.cc:648] HybridClock initialized: now 1790534177315955 us; error 54 us; skew 500 ppm
I20260927 18:36:17.316337 16731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:17 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:17.319705 16731 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:36:17.321091 16731 webserver.cc:460] Webserver started at http://127.0.97.67:35943/ using document root <none> and password file <none>
I20260927 18:36:17.321758 16731 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:17.321929 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:17.327191 16731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:36:17.329690 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:17.331009 16731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:36:17.331176 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "628aa59f1faa4072bb345be3dad4b782"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "18c55d44a815563ead76eb05e06efc1e"
server_key_iv: "78ab8dce5365ee2ce1cad58532fbb8c4"
server_key_version: "encryptionkey@0"
I20260927 18:36:17.332134 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:17.345511 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:17.349321 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:17.349586 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:17.350555 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:17.351963 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:17.352047 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:17.352180 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:17.352269 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:17.389839 16731 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44951
I20260927 18:36:17.389958 16858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44951 every 8 connection(s)
I20260927 18:36:17.391377 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:17.399659 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16731
Sep 27 18:36:17 dist-test-slave-n3p4 krb5kdc[15826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:17.414433 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.393997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46395 (local address 127.0.97.126:42279)
0927 18:36:17.394202 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.394209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.395442 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 18:36:17.397246 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.397252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.397344 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.397462 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:17.399035 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.399741 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.402169 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.402625 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.402718 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.406827 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:17.406834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:17.406838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:17.406879 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:17.410709 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.411691 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.411695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.411698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.411824 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.412338 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.412341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.412343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.412688 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:17.412847 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.413719 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.414041 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:36:17.415426 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.394561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:46395)
0927 18:36:17.395277 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.395313 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.396977 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.397707 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.397716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.398288 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:17.398837 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.398854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.399962 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.399968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.402034 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.402051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.402851 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.402860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.403178 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.403940 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:17.403962 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:17.406594 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:17.410837 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:17.410847 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:17.410869 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:17.411564 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:17.411946 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:17.411954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:17.411959 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:17.412211 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:17.412845 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:17.412855 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:17.413321 (+ 466us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.413634 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.414107 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260927 18:36:17.416600 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.400665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.97.126:42279)
0927 18:36:17.401149 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.401155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.401482 (+ 327us) server_negotiation.cc:407] Connection header received
0927 18:36:17.401639 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.401643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.401693 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.401773 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:17.403588 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.404390 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.406167 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.406860 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.406965 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.410486 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:17.410491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:17.410495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:17.410527 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:17.413675 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.414509 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.414514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.414518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.414645 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.415010 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.415015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.415017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.415324 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:17.415589 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.416082 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.416286 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260927 18:36:17.416942 16859 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42279
I20260927 18:36:17.417248 16859 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:17.418018 16859 heartbeater.cc:511] Master 127.0.97.126:42279 requested a full tablet report, sending...
I20260927 18:36:17.419445 16411 ts_manager.cc:194] Registered new tserver with Master: 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951)
I20260927 18:36:17.420202 16411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:46395
I20260927 18:36:17.431106 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:17.437916 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.431943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.0.97.126:42279)
0927 18:36:17.432147 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.432152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.432609 (+ 457us) server_negotiation.cc:407] Connection header received
0927 18:36:17.432784 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.432787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.432921 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.433090 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:17.433535 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.434495 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.435978 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.436479 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.436573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.437142 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:17.437360 (+ 218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:36:17.437646 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43208: token expired
Metrics: {"server-negotiator.queue_time_us":98}
W20260927 18:36:17.438076 16862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43208: token expired
W20260927 18:36:17.438476 16333 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.126:42279)
I20260927 18:36:17.451234 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.439050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.0.97.126:42279)
0927 18:36:17.439208 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.439215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.439236 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:36:17.439283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.439290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.439345 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.439450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:17.440762 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.441509 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.443051 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.443447 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.443580 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.446214 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:17.446226 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:17.446229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:17.446264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:17.448620 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.449269 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.449275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.449278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.449406 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.449761 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.449767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.449770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.450109 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:17.450362 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.450665 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.450878 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:36:17.454803 16411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:36:17.462184 16411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:36:17.500787 16596 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:36:17.504552 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.475336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.97.66:39623)
0927 18:36:17.494337 (+ 19001us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.494347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.495225 (+ 878us) server_negotiation.cc:407] Connection header received
0927 18:36:17.495344 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.495353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.495685 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.495898 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:17.496918 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.498513 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.501992 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.503339 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.503495 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.503695 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.504076 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.504233 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18811,"thread_start_us":18675,"threads_started":1}
I20260927 18:36:17.504626 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.474662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39623 (local address 127.0.0.1:44874)
0927 18:36:17.494871 (+ 20209us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.494897 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.495064 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.496209 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.496217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.496244 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:17.496708 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.496727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.498671 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.498676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.501812 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.501821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.503583 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.503589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.503790 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.503831 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.503970 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.504155 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20057,"mutex_wait_us":258,"thread_start_us":19906,"threads_started":1}
I20260927 18:36:17.505234 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.474549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.97.67:44951)
0927 18:36:17.495902 (+ 21353us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.495911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.495942 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:36:17.496064 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.496071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.496313 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.496514 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:17.497268 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.498844 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.501965 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.503647 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.503765 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.503978 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.504727 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.504883 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21171,"thread_start_us":21019,"threads_started":1}
I20260927 18:36:17.506453 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.474993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33459 (local address 127.0.0.1:40064)
0927 18:36:17.495241 (+ 20248us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.495258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.495347 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.497620 (+ 2273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.497625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.497648 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:17.498008 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.498022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.500100 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.500103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.503420 (+ 3317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.503434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.505272 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.505280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.505527 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.505558 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.505644 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.505806 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20123,"mutex_wait_us":5,"thread_start_us":19972,"threads_started":1}
I20260927 18:36:17.506747 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.474380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44951 (local address 127.0.0.1:60840)
0927 18:36:17.489491 (+ 15111us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.489570 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.490610 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.496524 (+ 5914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.496532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.496583 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:17.497074 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.497087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.498994 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.499005 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.501801 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.501816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.504140 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.504147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.504318 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.504351 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.504435 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.504586 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14743,"mutex_wait_us":113,"thread_start_us":15415,"threads_started":1}
I20260927 18:36:17.509022 16663 tablet_service.cc:1467] Processing CreateTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b (DEFAULT_TABLE table=test-table [id=04b5f17c2b794147ba352df26c328fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:17.510288 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.475601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.0.97.65:33459)
0927 18:36:17.497087 (+ 21486us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.497096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.497124 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:36:17.497223 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.497231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.497412 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.497634 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:17.498176 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.499914 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.503611 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.505060 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.505187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.505412 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.505762 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.505977 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21285,"thread_start_us":22231,"threads_started":1}
I20260927 18:36:17.510520 16663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a495d5daf3437ba9badebbd0cc8d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:17.510160 16794 tablet_service.cc:1467] Processing CreateTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b (DEFAULT_TABLE table=test-table [id=04b5f17c2b794147ba352df26c328fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:17.511476 16794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a495d5daf3437ba9badebbd0cc8d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:17.513085 16531 tablet_service.cc:1467] Processing CreateTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b (DEFAULT_TABLE table=test-table [id=04b5f17c2b794147ba352df26c328fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:17.514287 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a495d5daf3437ba9badebbd0cc8d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:17.525676 16875 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Bootstrap starting.
I20260927 18:36:17.528959 16875 tablet_bootstrap.cc:654] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:17.529657 16876 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Bootstrap starting.
I20260927 18:36:17.530440 16875 log.cc:824] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:17.532287 16877 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Bootstrap starting.
I20260927 18:36:17.534144 16876 tablet_bootstrap.cc:654] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:17.535279 16875 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: No bootstrap required, opened a new log
I20260927 18:36:17.535364 16876 log.cc:824] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:17.535610 16875 ts_tablet_manager.cc:1397] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 18:36:17.535638 16877 tablet_bootstrap.cc:654] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:17.537012 16877 log.cc:824] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:17.543157 16877 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: No bootstrap required, opened a new log
I20260927 18:36:17.543160 16876 tablet_bootstrap.cc:492] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: No bootstrap required, opened a new log
I20260927 18:36:17.543416 16877 ts_tablet_manager.cc:1397] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260927 18:36:17.543416 16876 ts_tablet_manager.cc:1397] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 18:36:17.543691 16875 raft_consensus.cc:348] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.544116 16875 raft_consensus.cc:374] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:17.544227 16875 raft_consensus.cc:729] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec6d338f884b1aa1e9fbca1c95477e, State: Initialized, Role: FOLLOWER
I20260927 18:36:17.544812 16875 consensus_queue.cc:260] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.546548 16875 ts_tablet_manager.cc:1428] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 18:36:17.554177 16877 raft_consensus.cc:348] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.554177 16876 raft_consensus.cc:348] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.554606 16876 raft_consensus.cc:374] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:17.554605 16877 raft_consensus.cc:374] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:17.554708 16876 raft_consensus.cc:729] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f24691d17f4a3d82da9053fb274390, State: Initialized, Role: FOLLOWER
I20260927 18:36:17.554708 16877 raft_consensus.cc:729] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628aa59f1faa4072bb345be3dad4b782, State: Initialized, Role: FOLLOWER
I20260927 18:36:17.555322 16877 consensus_queue.cc:260] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.555322 16876 consensus_queue.cc:260] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.556428 16859 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:36:17.557008 16728 heartbeater.cc:503] Master 127.0.97.126:42279 was elected leader, sending a full tablet report...
I20260927 18:36:17.557176 16877 ts_tablet_manager.cc:1428] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260927 18:36:17.557468 16876 ts_tablet_manager.cc:1428] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260927 18:36:17.643049 16860 tablet.cc:2369] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:36:17.685648 16729 tablet.cc:2369] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:36:17.717278 16597 tablet.cc:2369] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:17.784643 16881 raft_consensus.cc:482] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:17.784996 16881 raft_consensus.cc:504] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.786798 16881 leader_election.cc:288] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951), 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623)
I20260927 18:36:17.794994 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.787324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55245 (local address 127.0.97.67:44951)
0927 18:36:17.787543 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.787550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.788429 (+ 879us) server_negotiation.cc:407] Connection header received
0927 18:36:17.788579 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.788583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.788642 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.788708 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:17.789373 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.790537 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.793008 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.794043 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.794167 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.794262 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.794628 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.794753 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:36:17.795186 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.787431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44951 (local address 127.0.97.65:55245)
0927 18:36:17.788326 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.788340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.788442 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.788756 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.788761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.788782 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:17.789180 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.789188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.790729 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.790733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.792866 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.792877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.794263 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.794272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.794441 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.794485 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.794587 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.794735 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":53,"thread_start_us":481,"threads_started":1}
I20260927 18:36:17.795302 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.787355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:51421 (local address 127.0.97.66:39623)
0927 18:36:17.787564 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.787571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.788331 (+ 760us) server_negotiation.cc:407] Connection header received
0927 18:36:17.788600 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.788604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.788661 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.788763 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:17.789488 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.790549 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.793016 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.794177 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.794311 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.794403 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.794924 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.795082 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:36:17.795754 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.787372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39623 (local address 127.0.97.65:51421)
0927 18:36:17.788215 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:17.788234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:17.788407 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:17.788807 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:17.788811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:17.788837 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:17.789346 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.789359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.790680 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.790684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.792863 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:17.792876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.794549 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.794557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:17.794755 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.794786 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:36:17.794883 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:36:17.795264 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":76,"thread_start_us":522,"threads_started":1}
I20260927 18:36:17.796558 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a495d5daf3437ba9badebbd0cc8d5b" candidate_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aa59f1faa4072bb345be3dad4b782" is_pre_election: true
I20260927 18:36:17.796788 16683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a495d5daf3437ba9badebbd0cc8d5b" candidate_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f24691d17f4a3d82da9053fb274390" is_pre_election: true
I20260927 18:36:17.797183 16814 raft_consensus.cc:2454] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec6d338f884b1aa1e9fbca1c95477e in term 0.
I20260927 18:36:17.797262 16683 raft_consensus.cc:2454] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec6d338f884b1aa1e9fbca1c95477e in term 0.
I20260927 18:36:17.797963 16487 leader_election.cc:302] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628aa59f1faa4072bb345be3dad4b782, 77ec6d338f884b1aa1e9fbca1c95477e; no voters:
I20260927 18:36:17.798370 16881 raft_consensus.cc:2781] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:17.798540 16881 raft_consensus.cc:482] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:17.798662 16881 raft_consensus.cc:3037] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:17.800864 16881 raft_consensus.cc:504] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.801474 16881 leader_election.cc:288] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [CANDIDATE]: Term 1 election: Requested vote from peers 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951), 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623)
I20260927 18:36:17.802053 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a495d5daf3437ba9badebbd0cc8d5b" candidate_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aa59f1faa4072bb345be3dad4b782"
I20260927 18:36:17.802086 16683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a495d5daf3437ba9badebbd0cc8d5b" candidate_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f24691d17f4a3d82da9053fb274390"
I20260927 18:36:17.802284 16683 raft_consensus.cc:3037] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:17.802284 16814 raft_consensus.cc:3037] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:17.805553 16683 raft_consensus.cc:2454] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec6d338f884b1aa1e9fbca1c95477e in term 1.
I20260927 18:36:17.805560 16814 raft_consensus.cc:2454] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec6d338f884b1aa1e9fbca1c95477e in term 1.
I20260927 18:36:17.806052 16485 leader_election.cc:302] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57f24691d17f4a3d82da9053fb274390, 628aa59f1faa4072bb345be3dad4b782, 77ec6d338f884b1aa1e9fbca1c95477e; no voters:
I20260927 18:36:17.806391 16881 raft_consensus.cc:2781] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:17.806911 16881 raft_consensus.cc:686] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 LEADER]: Becoming Leader. State: Replica: 77ec6d338f884b1aa1e9fbca1c95477e, State: Running, Role: LEADER
I20260927 18:36:17.807432 16881 consensus_queue.cc:237] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } }
I20260927 18:36:17.811678 16411 catalog_manager.cc:5495] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65). New cstate: current_term: 1 leader_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ec6d338f884b1aa1e9fbca1c95477e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33459 } health_report { overall_health: HEALTHY } } }
I20260927 18:36:17.878979 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.873165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.97.65:33459)
0927 18:36:17.873447 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.873454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.873580 (+ 126us) server_negotiation.cc:407] Connection header received
0927 18:36:17.873872 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.873876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.874000 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.874170 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:17.874566 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.875294 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.876677 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.877113 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.877253 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.877845 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:17.878392 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:17.878561 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.878617 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.878699 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 18:36:17.887279 16814 raft_consensus.cc:1261] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Refusing update from remote peer 77ec6d338f884b1aa1e9fbca1c95477e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:17.887276 16683 raft_consensus.cc:1261] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Refusing update from remote peer 77ec6d338f884b1aa1e9fbca1c95477e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:17.888263 16886 consensus_queue.cc:1035] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [LEADER]: Connected to new peer: Peer: permanent_uuid: "628aa59f1faa4072bb345be3dad4b782" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:17.888679 16881 consensus_queue.cc:1035] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f24691d17f4a3d82da9053fb274390" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:17.913118 16410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:17.913463 16410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:17.917697 16410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Sending DeleteTablet for 3 replicas of tablet b3a495d5daf3437ba9badebbd0cc8d5b
I20260927 18:36:17.919003 16794 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:60840
I20260927 18:36:17.919126 16663 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44874
I20260927 18:36:17.919114 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a495d5daf3437ba9badebbd0cc8d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40064
I20260927 18:36:17.923797 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16448
I20260927 18:36:17.934841 16901 tablet_replica.cc:330] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: stopping tablet replica
I20260927 18:36:17.935314 16901 raft_consensus.cc:2229] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:36:17.935981 16901 raft_consensus.cc:2258] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:17.936806 16900 tablet_replica.cc:330] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: stopping tablet replica
I20260927 18:36:17.937373 16900 raft_consensus.cc:2229] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:17.937901 16900 raft_consensus.cc:2258] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:17.938866 16899 tablet_replica.cc:330] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: stopping tablet replica
I20260927 18:36:17.939239 16899 raft_consensus.cc:2229] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:17.939695 16899 raft_consensus.cc:2258] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:17.939940 16901 ts_tablet_manager.cc:1905] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:17.940405 16900 ts_tablet_manager.cc:1905] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:17.944195 16899 ts_tablet_manager.cc:1905] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:17.947851 16901 ts_tablet_manager.cc:1918] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:17.948040 16901 log.cc:1196] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/b3a495d5daf3437ba9badebbd0cc8d5b
I20260927 18:36:17.948563 16901 ts_tablet_manager.cc:1939] T b3a495d5daf3437ba9badebbd0cc8d5b P 77ec6d338f884b1aa1e9fbca1c95477e: Deleting consensus metadata
I20260927 18:36:17.949172 16900 ts_tablet_manager.cc:1918] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:17.949397 16900 log.cc:1196] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/b3a495d5daf3437ba9badebbd0cc8d5b
I20260927 18:36:17.949991 16900 ts_tablet_manager.cc:1939] T b3a495d5daf3437ba9badebbd0cc8d5b P 57f24691d17f4a3d82da9053fb274390: Deleting consensus metadata
I20260927 18:36:17.950438 16395 catalog_manager.cc:4841] TS 77ec6d338f884b1aa1e9fbca1c95477e (127.0.97.65:33459): tablet b3a495d5daf3437ba9badebbd0cc8d5b (table test-table [id=04b5f17c2b794147ba352df26c328fc9]) successfully deleted
I20260927 18:36:17.952090 16396 catalog_manager.cc:4841] TS 57f24691d17f4a3d82da9053fb274390 (127.0.97.66:39623): tablet b3a495d5daf3437ba9badebbd0cc8d5b (table test-table [id=04b5f17c2b794147ba352df26c328fc9]) successfully deleted
I20260927 18:36:17.953192 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:17.925105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.0.97.65:33459)
0927 18:36:17.925322 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:17.925331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:17.929512 (+ 4181us) server_negotiation.cc:407] Connection header received
0927 18:36:17.929742 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:17.929750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:17.929887 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:17.929997 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:17.934914 (+ 4917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.936042 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.940059 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:17.940778 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:17.940874 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:17.944835 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:17.944866 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:17.944885 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:17.944934 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:17.949145 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.950130 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.950138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.950141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.950323 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:17.950811 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:17.950815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:17.950819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:17.951276 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:17.951590 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:17.952604 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0927 18:36:17.952841 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:36:17.953735 16899 ts_tablet_manager.cc:1918] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:17.954245 16899 log.cc:1196] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/b3a495d5daf3437ba9badebbd0cc8d5b
I20260927 18:36:17.954668 16591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:17.954886 16899 ts_tablet_manager.cc:1939] T b3a495d5daf3437ba9badebbd0cc8d5b P 628aa59f1faa4072bb345be3dad4b782: Deleting consensus metadata
I20260927 18:36:17.956990 16398 catalog_manager.cc:4841] TS 628aa59f1faa4072bb345be3dad4b782 (127.0.97.67:44951): tablet b3a495d5daf3437ba9badebbd0cc8d5b (table test-table [id=04b5f17c2b794147ba352df26c328fc9]) successfully deleted
I20260927 18:36:18.053337 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16448
I20260927 18:36:18.073172 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16600
I20260927 18:36:18.086881 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:18.073954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.0.97.66:39623)
0927 18:36:18.074238 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:18.074244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:18.074343 (+ 99us) server_negotiation.cc:407] Connection header received
0927 18:36:18.074564 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:18.074576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:18.074652 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:18.074756 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:18.076048 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:18.076764 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:18.078164 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:18.078579 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:18.078714 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:18.081239 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:18.081253 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:18.081269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:18.081301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:18.084064 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:18.084745 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:18.084753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:18.084755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:18.084883 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:18.085264 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:18.085271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:18.085274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:18.085665 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:18.085997 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:18.086319 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:36:18.086546 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260927 18:36:18.087549 16723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:18.176347 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16600
I20260927 18:36:18.193722 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16731
I20260927 18:36:18.208034 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:18.194601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.97.67:44951)
0927 18:36:18.194771 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:18.194776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:18.194844 (+ 68us) server_negotiation.cc:407] Connection header received
0927 18:36:18.195063 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:18.195069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:18.195147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:18.195234 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:18.196819 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:18.197515 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:18.199064 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:18.199432 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:18.199524 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:18.202348 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:18.202364 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:18.202380 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:18.202416 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:18.205440 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:18.206252 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:18.206259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:18.206262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:18.206427 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:18.206764 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:18.206767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:18.206770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:18.207115 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:18.207327 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:18.207628 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:36:18.207786 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:36:18.208680 16854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:18.302098 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16731
I20260927 18:36:18.322520 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16378
I20260927 18:36:18.323632 16441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:18.438813 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16378
I20260927 18:36:38.460877 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--webserver_interface=127.0.97.126
--webserver_port=38047
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:38.791836 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:38.792268 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:38.792447 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:38.799809 16907 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:36:38.799952 16907 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:36:38.800050 16907 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:36:38.800135 16907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:38.800244 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:38.800354 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:38.800455 16907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:38.800545 16907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:38.810520 16907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:38.811841 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:38.813247 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:38.825932 16912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:38.825940 16915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:38.825968 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:38.826892 16907 server_base.cc:1034] running on GCE node
I20260927 18:36:38.827626 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:38.828962 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:38.830206 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790534198830141 us; error 39 us; skew 500 ppm
I20260927 18:36:38.830574 16907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:38 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:38.839664 16907 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:36:38.841341 16907 webserver.cc:460] Webserver started at http://127.0.97.126:38047/ using document root <none> and password file <none>
I20260927 18:36:38.842150 16907 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:38.842281 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:38.847750 16907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:36:38.850466 16922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:38.851765 16907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:38.851944 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5d9508c1d9214ae9bfa8f499aa7810a3"
format_stamp: "Formatted at 2026-09-27 18:35:52 on dist-test-slave-n3p4"
server_key: "4708aad9c3596e2172c466bb821719eb"
server_key_iv: "3d28e0b12590ea62ab79744e3c3873c5"
server_key_version: "encryptionkey@0"
I20260927 18:36:38.852905 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:38.895553 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:38.899809 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:38.900143 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:38.919296 16907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42279
I20260927 18:36:38.919337 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42279 every 8 connection(s)
I20260927 18:36:38.920881 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:38.924656 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16907
I20260927 18:36:38.925631 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:33459
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=45691
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:38.929378 16974 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:36:38.934561 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap starting.
I20260927 18:36:38.954483 16974 log.cc:824] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:38.970314 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:38.970940 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Bootstrap complete.
I20260927 18:36:38.979938 16974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:38.980826 16974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Initialized, Role: FOLLOWER
I20260927 18:36:38.981405 16974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:38.981653 16974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:38.981779 16974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:38.982030 16974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:36:38.985911 16974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:38.986400 16974 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9508c1d9214ae9bfa8f499aa7810a3; no voters:
I20260927 18:36:38.987076 16974 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:36:38.987228 16978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:36:38.988441 16978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [term 3 LEADER]: Becoming Leader. State: Replica: 5d9508c1d9214ae9bfa8f499aa7810a3, State: Running, Role: LEADER
I20260927 18:36:38.989054 16978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } }
I20260927 18:36:38.989312 16974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:38.991603 16980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9508c1d9214ae9bfa8f499aa7810a3. Latest consensus state: current_term: 3 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:36:38.991973 16980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:38.992057 16979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9508c1d9214ae9bfa8f499aa7810a3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42279 } } }
I20260927 18:36:38.992440 16979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:38.994388 16986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:39.008157 16986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:39.009893 16986 catalog_manager.cc:1259] Loaded cluster ID: 30e543dcc3204265aac8686c757a5d7d
I20260927 18:36:39.013937 16986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:39.019905 16986 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:39.022097 16986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Loaded TSK: 0
I20260927 18:36:39.022306 16986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d9508c1d9214ae9bfa8f499aa7810a3: Loaded TSK: 1
F20260927 18:36:39.023748 16986 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7b4fd1a9ad google::LogMessage::Fail() at ??:0
@ 0x7f7b4fd1d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7b4fd1a4de google::LogMessage::Flush() at ??:0
@ 0x7f7b4fd1ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7b53a7e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7b53a6d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7b657491c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7b65748783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7b657d42a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7b657d40c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7b6581e5e9 std::function<>::operator()() at ??:0
@ 0x7f7b6574ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7b6574ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7b657d2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7b657d2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b52966345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7b5296991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7b52969799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b52944eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7b55c1e6db start_thread at ??:0
@ 0x7f7b4ebb461f clone at ??:0
W20260927 18:36:39.339972 16976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:39.340276 16976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:39.340369 16976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:39.347724 16976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:39.347836 16976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:39.347949 16976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:36:39.360484 16976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:33459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=45691
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:39.362109 16976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:39.363459 16976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:39.376539 17021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:39.376622 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:39.376493 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:39.377605 16976 server_base.cc:1034] running on GCE node
I20260927 18:36:39.378293 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:39.379208 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:39.380476 16976 hybrid_clock.cc:648] HybridClock initialized: now 1790534199380430 us; error 50 us; skew 500 ppm
I20260927 18:36:39.380815 16976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:39 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:39.389417 16976 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:36:39.391018 16976 webserver.cc:460] Webserver started at http://127.0.97.65:45691/ using document root <none> and password file <none>
I20260927 18:36:39.391698 16976 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:39.391845 16976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:39.397374 16976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:36:39.400184 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.401510 16976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:36:39.401698 16976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "77ec6d338f884b1aa1e9fbca1c95477e"
format_stamp: "Formatted at 2026-09-27 18:35:53 on dist-test-slave-n3p4"
server_key: "f78837b51941dc20b9b41c1f800556d6"
server_key_iv: "066081d0ae2ad951ba7c5856359a31f4"
server_key_version: "encryptionkey@0"
I20260927 18:36:39.402761 16976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:39.420636 16976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:39.424510 16976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:39.424770 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:39.425729 16976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:39.427016 16976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:39.427142 16976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.427280 16976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:39.427368 16976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.473335 16976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33459
I20260927 18:36:39.473392 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33459 every 8 connection(s)
I20260927 18:36:39.475128 16976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:39.481184 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.479221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:48589)
0927 18:36:39.480077 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.480289 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
W20260927 18:36:39.481966 17144 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:42279: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
I20260927 18:36:39.483057 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.482566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:53399)
0927 18:36:39.482791 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.482890 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 18:36:39.483809 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 16976
I20260927 18:36:39.484076 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.483681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:60639)
0927 18:36:39.483840 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.483941 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:36:39.484321 17144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:36:39.484608 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:39623
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=38983
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:39.813660 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:39.813992 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:39.814097 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:39.821053 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:39.821146 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:39.821251 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:36:39.830734 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:39623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=38983
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:39.831866 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:39.833083 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:39.846022 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:39.846082 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:39.846604 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:39.846791 17147 server_base.cc:1034] running on GCE node
I20260927 18:36:39.847432 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:39.848481 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:39.849682 17147 hybrid_clock.cc:648] HybridClock initialized: now 1790534199849644 us; error 37 us; skew 500 ppm
I20260927 18:36:39.850021 17147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:39 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:39.853170 17147 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:36:39.854583 17147 webserver.cc:460] Webserver started at http://127.0.97.66:38983/ using document root <none> and password file <none>
I20260927 18:36:39.855225 17147 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:39.855324 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:39.860782 17147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:36:39.863320 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.864584 17147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 18:36:39.864759 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "57f24691d17f4a3d82da9053fb274390"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "3d4725454ca0c6c18d35d0e858575d3a"
server_key_iv: "d7e41257819463f98e683c3b8291e891"
server_key_version: "encryptionkey@0"
I20260927 18:36:39.865694 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:39.883224 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:39.887794 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:39.888087 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:39.889151 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:39.890468 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:39.890565 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.890731 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:39.890822 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:39.924847 17147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39623
I20260927 18:36:39.924935 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39623 every 8 connection(s)
I20260927 18:36:39.926333 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:39.929561 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17147
I20260927 18:36:39.930397 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:44951
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:42279
--webserver_port=35943
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:39.931334 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.929323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:53857)
0927 18:36:39.930186 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.930367 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":407,"threads_started":1}
W20260927 18:36:39.931988 17275 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:42279: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
I20260927 18:36:39.932778 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.932464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:55959)
0927 18:36:39.932611 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.932680 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:36:39.933748 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:39.933306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:42999)
0927 18:36:39.933486 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:39.933588 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:36:39.934052 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:36:40.246578 17278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:40.246836 17278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:40.247030 17278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:40.254158 17278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:40.254292 17278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:40.254477 17278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:36:40.263955 17278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:44951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=35943
--tserver_master_addrs=127.0.97.126:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:40.265208 17278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:40.266520 17278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:40.279330 17283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:40.279268 17286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:40.279381 17284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:40.280412 17278 server_base.cc:1034] running on GCE node
I20260927 18:36:40.281116 17278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:40.282130 17278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:40.283416 17278 hybrid_clock.cc:648] HybridClock initialized: now 1790534200283337 us; error 42 us; skew 500 ppm
I20260927 18:36:40.283751 17278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:40 dist-test-slave-n3p4 krb5kdc[15826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:40.286780 17278 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:36:40.288168 17278 webserver.cc:460] Webserver started at http://127.0.97.67:35943/ using document root <none> and password file <none>
I20260927 18:36:40.288829 17278 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:40.288971 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:40.294318 17278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:40.296984 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:40.298318 17278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:40.298487 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "628aa59f1faa4072bb345be3dad4b782"
format_stamp: "Formatted at 2026-09-27 18:35:54 on dist-test-slave-n3p4"
server_key: "18c55d44a815563ead76eb05e06efc1e"
server_key_iv: "78ab8dce5365ee2ce1cad58532fbb8c4"
server_key_version: "encryptionkey@0"
I20260927 18:36:40.299444 17278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:40.313640 17278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:40.317529 17278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:40.317816 17278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:40.318737 17278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:40.319977 17278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:40.320101 17278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:40.320236 17278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:40.320325 17278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:40.353703 17278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44951
I20260927 18:36:40.353842 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44951 every 8 connection(s)
I20260927 18:36:40.355243 17278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:40.360101 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.358368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:45217)
0927 18:36:40.359055 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:40.359256 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
W20260927 18:36:40.360721 17406 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:42279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:42279: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
I20260927 18:36:40.361466 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.361186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:40791)
0927 18:36:40.361307 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:40.361370 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 18:36:40.362072 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17278
I20260927 18:36:40.362421 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.362096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:57991)
0927 18:36:40.362255 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:40.362322 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 18:36:40.362818 17406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:36:40.485613 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.484785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:44095)
0927 18:36:40.485299 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:40.485391 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":292,"threads_started":1}
I20260927 18:36:40.935497 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.934546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:49871)
0927 18:36:40.935012 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:40.935119 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260927 18:36:41.364080 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.363305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:48319)
0927 18:36:41.363755 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:41.363854 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260927 18:36:41.487169 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.486430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:57029)
0927 18:36:41.486903 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:41.486995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260927 18:36:41.937136 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.936442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:36919)
0927 18:36:41.936905 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:41.936984 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260927 18:36:42.365540 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.364887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:57705)
0927 18:36:42.365311 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.365385 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260927 18:36:42.488793 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.488087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:56681)
0927 18:36:42.488536 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.488620 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260927 18:36:42.938876 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.938169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:38389)
0927 18:36:42.938619 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.938703 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260927 18:36:43.367202 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.366353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:34145)
0927 18:36:43.366816 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.366917 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":258,"threads_started":1}
I20260927 18:36:43.490533 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.489705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:52341)
0927 18:36:43.490270 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.490358 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":302,"threads_started":1}
I20260927 18:36:43.940336 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.939671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:50541)
0927 18:36:43.940077 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.940150 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260927 18:36:44.368765 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.368082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:47243)
0927 18:36:44.368485 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.368583 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":198,"threads_started":1}
I20260927 18:36:44.492327 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.491514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:55167)
0927 18:36:44.492013 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.492113 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260927 18:36:44.942039 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.941239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:34351)
0927 18:36:44.941716 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.941851 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260927 18:36:45.370415 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.369635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:47509)
0927 18:36:45.370172 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.370253 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260927 18:36:45.493965 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.493201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:43109)
0927 18:36:45.493650 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.493737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260927 18:36:45.943734 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.942978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:44243)
0927 18:36:45.943454 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.943576 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260927 18:36:46.371903 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.371230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:45295)
0927 18:36:46.371662 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.371747 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260927 18:36:46.496004 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.495014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:51977)
0927 18:36:46.495580 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.495695 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":285,"threads_started":1}
I20260927 18:36:46.945437 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.944593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:32903)
0927 18:36:46.945110 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.945206 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":235,"threads_started":1}
I20260927 18:36:47.373656 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.372786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:47021)
0927 18:36:47.373329 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.373418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
I20260927 18:36:47.497900 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.497016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:36689)
0927 18:36:47.497558 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.497646 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":307,"threads_started":1}
I20260927 18:36:47.947042 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.946338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:42235)
0927 18:36:47.946803 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.946881 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260927 18:36:48.375663 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.374719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:48659)
0927 18:36:48.375382 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.375478 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":299,"threads_started":1}
I20260927 18:36:48.499576 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.498856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:46125)
0927 18:36:48.499319 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.499403 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":216,"threads_started":1}
I20260927 18:36:48.948583 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.947883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:41185)
0927 18:36:48.948341 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.948419 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260927 18:36:49.377285 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.376562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:41059)
0927 18:36:49.377030 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.377117 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":229,"threads_started":1}
I20260927 18:36:49.501657 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.500558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:35587)
0927 18:36:49.501268 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.501401 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":298,"threads_started":1}
I20260927 18:36:49.950345 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.949600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:48947)
0927 18:36:49.950103 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.950186 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":308,"threads_started":1}
I20260927 18:36:50.378986 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.378321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:40431)
0927 18:36:50.378763 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.378838 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":256,"threads_started":1}
I20260927 18:36:50.503335 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.502610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:45313)
0927 18:36:50.503072 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.503166 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":208,"threads_started":1}
I20260927 18:36:50.951932 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.951201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:60909)
0927 18:36:50.951663 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.951749 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260927 18:36:51.380649 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.379937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:35423)
0927 18:36:51.380376 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.380467 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":213,"threads_started":1}
I20260927 18:36:51.504908 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.504195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:37031)
0927 18:36:51.504616 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.504697 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":208,"threads_started":1}
I20260927 18:36:51.953636 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.952919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:59413)
0927 18:36:51.953370 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.953457 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260927 18:36:52.382339 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.381575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:50501)
0927 18:36:52.382055 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.382137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":277,"threads_started":1}
I20260927 18:36:52.506800 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.505919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:57575)
0927 18:36:52.506479 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.506594 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":294,"threads_started":1}
I20260927 18:36:52.955410 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.954627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:51027)
0927 18:36:52.955108 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.955202 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":232,"threads_started":1}
I20260927 18:36:53.384020 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.383267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:45431)
0927 18:36:53.383741 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.383833 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260927 18:36:53.508469 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.507734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:47801)
0927 18:36:53.508212 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.508289 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260927 18:36:53.957208 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.956464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:58915)
0927 18:36:53.956932 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.957026 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260927 18:36:54.385850 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.385018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:51403)
0927 18:36:54.385492 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.385625 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260927 18:36:54.510360 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.509554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.65:49885)
0927 18:36:54.510078 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.510172 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":322,"threads_started":1}
I20260927 18:36:54.959064 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.958313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.66:49415)
0927 18:36:54.958784 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.958875 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":218,"threads_started":1}
I20260927 18:36:55.368541 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 16976
I20260927 18:36:55.387817 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.387052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42279 (local address 127.0.97.67:59827)
0927 18:36:55.387545 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:55.387635 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
I20260927 18:36:55.392220 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.369718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.97.65:33459)
0927 18:36:55.370442 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:55.370477 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:55.370521 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:36:55.372433 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:55.372442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:55.372868 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:55.373152 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:55.374679 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.376381 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.377760 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.378452 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.378582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:55.386272 (+ 7690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:55.386280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:55.386297 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:55.386342 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:55.389302 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.390037 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.390043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.390049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.390231 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.390549 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.390552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.390554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.391038 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:55.391270 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:55.391641 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 18:36:55.391951 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":376,"threads_started":1}
I20260927 18:36:55.393606 17139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:55.480984 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 16976
I20260927 18:36:55.498368 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17147
I20260927 18:36:55.516077 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.499373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.0.97.66:39623)
0927 18:36:55.499893 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:55.499908 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:55.499945 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:36:55.501494 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:55.501504 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:55.502012 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:55.502387 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:55.503872 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.505119 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.506485 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.507166 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.507357 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:55.509972 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:55.509979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:55.509993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:55.510033 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:55.512809 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.513523 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.513529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.513535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.513697 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.514108 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.514114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.514117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.514618 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:55.514920 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:55.515398 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 18:36:55.515794 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260927 18:36:55.517519 17270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:55.603968 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17147
I20260927 18:36:55.622042 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17278
I20260927 18:36:55.639966 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.623173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.97.67:44951)
0927 18:36:55.623744 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:55.623759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:55.623797 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:36:55.625371 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:55.625381 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:55.625872 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:55.626189 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:55.627572 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.628755 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.630284 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:55.630935 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:55.631067 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:55.633719 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:55.633727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:55.633741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:55.633775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:55.636743 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.637461 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.637467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.637473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.637645 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:55.638095 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:55.638098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:55.638100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:55.638600 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:55.638829 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:55.639253 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:36:55.639670 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
I20260927 18:36:55.641367 17401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:55.726977 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17278
2026-09-27T18:36:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64583 ms)
[----------] 30 tests from SecurityITest (296677 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:36:55 dist-test-slave-n3p4 krb5kdc[17463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:36:55 dist-test-slave-n3p4 krb5kdc[17463](info): set up 2 sockets
Sep 27 18:36:55 dist-test-slave-n3p4 krb5kdc[17463](info): commencing operation
krb5kdc: starting...
W20260927 18:36:56.837841 389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:36:56 dist-test-slave-n3p4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:36:57.086237 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45575
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:57.422677 17476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:57.422967 17476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:57.423122 17476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:57.430487 17476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:57.430598 17476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:57.430701 17476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:57.430806 17476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:57.430892 17476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:57.440795 17476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:57.442152 17476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:57.443464 17476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:57.455575 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:57.455600 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:57.455667 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:57.456480 17476 server_base.cc:1034] running on GCE node
I20260927 18:36:57.457444 17476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:57.459182 17476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:57.460407 17476 hybrid_clock.cc:648] HybridClock initialized: now 1790534217460357 us; error 34 us; skew 500 ppm
I20260927 18:36:57.460843 17476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:57 dist-test-slave-n3p4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:57.464331 17476 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:36:57.466192 17476 webserver.cc:460] Webserver started at http://127.0.97.126:42195/ using document root <none> and password file <none>
I20260927 18:36:57.467227 17476 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:57.467374 17476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:57.467732 17476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:57.471477 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "5ddc59a34f294530be981409c4676621"
format_stamp: "Formatted at 2026-09-27 18:36:57 on dist-test-slave-n3p4"
server_key: "be4d59528721aed65d10461699cbf40e"
server_key_iv: "7449d750909e2f39b21733c553f5e065"
server_key_version: "encryptionkey@0"
I20260927 18:36:57.472338 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "5ddc59a34f294530be981409c4676621"
format_stamp: "Formatted at 2026-09-27 18:36:57 on dist-test-slave-n3p4"
server_key: "be4d59528721aed65d10461699cbf40e"
server_key_iv: "7449d750909e2f39b21733c553f5e065"
server_key_version: "encryptionkey@0"
I20260927 18:36:57.478080 17476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:36:57.481868 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:57.483227 17476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:36:57.483423 17476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5ddc59a34f294530be981409c4676621"
format_stamp: "Formatted at 2026-09-27 18:36:57 on dist-test-slave-n3p4"
server_key: "be4d59528721aed65d10461699cbf40e"
server_key_iv: "7449d750909e2f39b21733c553f5e065"
server_key_version: "encryptionkey@0"
I20260927 18:36:57.483632 17476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:57.504894 17476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:57.510243 17476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:57.510563 17476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:57.532883 17476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45575
I20260927 18:36:57.532930 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45575 every 8 connection(s)
I20260927 18:36:57.534495 17476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:57.538985 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:57.539665 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17476
I20260927 18:36:57.539863 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:36:57.540483 389 external_mini_cluster.cc:1428] Setting key 94677378ad0b84fc773a6c3cb3e1de24
I20260927 18:36:57.553256 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: Bootstrap starting.
Sep 27 18:36:57 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:57.556393 17543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:57.557687 17543 log.cc:824] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:57.561187 17543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: No bootstrap required, opened a new log
I20260927 18:36:57.565412 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:57.542552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.0.97.126:45575)
0927 18:36:57.543312 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:57.543329 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:57.543389 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:36:57.545650 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:57.545678 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:57.546211 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:57.546706 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:57.548050 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:57.550085 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:57.551474 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:57.552492 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:57.552685 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:57.556943 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:57.556953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:57.556970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:57.557019 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:57.561041 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:57.561741 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:57.561755 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:57.561763 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:57.561997 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:57.562371 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:57.562375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:57.562378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:57.563045 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:57.563350 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:57.563660 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 18:36:57.564104 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":326,"threads_started":1}
I20260927 18:36:57.570248 17543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } }
I20260927 18:36:57.570554 17543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:57.570623 17543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddc59a34f294530be981409c4676621, State: Initialized, Role: FOLLOWER
I20260927 18:36:57.571077 17543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } }
I20260927 18:36:57.571281 17543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:57.571375 17543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:57.571523 17543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:57.573678 17543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } }
I20260927 18:36:57.574222 17543 leader_election.cc:302] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ddc59a34f294530be981409c4676621; no voters:
I20260927 18:36:57.574843 17543 leader_election.cc:288] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:36:57.574935 17548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:57.575655 17548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [term 1 LEADER]: Becoming Leader. State: Replica: 5ddc59a34f294530be981409c4676621, State: Running, Role: LEADER
I20260927 18:36:57.576149 17548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } }
I20260927 18:36:57.576856 17543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:57.579156 17550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ddc59a34f294530be981409c4676621. Latest consensus state: current_term: 1 leader_uuid: "5ddc59a34f294530be981409c4676621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } } }
I20260927 18:36:57.579059 17549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ddc59a34f294530be981409c4676621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddc59a34f294530be981409c4676621" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45575 } } }
I20260927 18:36:57.579433 17550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:57.579433 17549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:57.580343 17554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:57.585986 17554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:57.595890 17554 catalog_manager.cc:1347] Generated new cluster ID: 1621f2a9b6d34fadab73b05956b988aa
I20260927 18:36:57.596020 17554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:57.620610 17554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:36:57.622066 17554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:57.634452 17554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: Generated new TSK 0
I20260927 18:36:57.635367 17554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:36:57.705682 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:58.055454 17569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:58.055806 17569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:58.055974 17569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:58.063339 17569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:58.063446 17569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:58.063649 17569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:36:58.073392 17569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:58.074759 17569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:58.076069 17569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:58.089289 17577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:58.089237 17575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:58.089329 17574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:58.090222 17569 server_base.cc:1034] running on GCE node
I20260927 18:36:58.090881 17569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:58.091782 17569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:58.092984 17569 hybrid_clock.cc:648] HybridClock initialized: now 1790534218092919 us; error 57 us; skew 500 ppm
I20260927 18:36:58.093313 17569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:58 dist-test-slave-n3p4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:58.096504 17569 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:36:58.097918 17569 webserver.cc:460] Webserver started at http://127.0.97.65:39291/ using document root <none> and password file <none>
I20260927 18:36:58.098575 17569 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:58.098718 17569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:58.099022 17569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:58.101835 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "f9a0b2ebe67141168247074c95736eb2"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "183d31b96d6c38d6d1773e2b7d5f13be"
server_key_iv: "dff4e119265fcd1572a8341fdc5bdab9"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.102646 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "f9a0b2ebe67141168247074c95736eb2"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "183d31b96d6c38d6d1773e2b7d5f13be"
server_key_iv: "dff4e119265fcd1572a8341fdc5bdab9"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.107508 17569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:36:58.110788 17584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.111959 17569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:36:58.112134 17569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "f9a0b2ebe67141168247074c95736eb2"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "183d31b96d6c38d6d1773e2b7d5f13be"
server_key_iv: "dff4e119265fcd1572a8341fdc5bdab9"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.112334 17569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:58.128333 17569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:58.132565 17569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:58.132850 17569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:58.133942 17569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:58.135362 17569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:58.135465 17569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.135612 17569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:58.135684 17569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.168282 17569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38169
I20260927 18:36:58.168324 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38169 every 8 connection(s)
I20260927 18:36:58.169556 17569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:58.173601 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17569
I20260927 18:36:58.173841 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:36:58.174520 389 external_mini_cluster.cc:1428] Setting key 32171b93474612fcfb5d140157753994
Sep 27 18:36:58 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:58.190140 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.172142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47313 (local address 127.0.97.126:45575)
0927 18:36:58.172611 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:58.172618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:58.173545 (+ 927us) server_negotiation.cc:407] Connection header received
0927 18:36:58.175593 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:58.175597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:58.175660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:58.175759 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:58.177368 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.178254 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.179754 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.180156 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.180292 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:58.183732 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:58.183737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:58.183740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:58.183778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:58.186781 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:58.187638 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:58.187649 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:58.187651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:58.187789 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:58.188243 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:58.188248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:58.188251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:58.188536 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:58.188720 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:58.189508 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 18:36:58.189739 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":220,"threads_started":1}
I20260927 18:36:58.191161 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.172701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45575 (local address 127.0.97.65:47313)
0927 18:36:58.173388 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:58.173425 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:58.175300 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:58.176035 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:58.176046 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:58.176591 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:58.177194 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:58.177213 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.178416 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.178420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:58.179607 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:58.179619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.180397 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.180404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:58.180630 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:58.181297 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:58.181319 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:58.183515 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:58.186933 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:58.186942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:58.186960 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:58.187405 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:58.187906 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:58.187911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:58.187915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:58.188125 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:58.188789 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:58.188795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:58.189127 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 18:36:58.189411 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 18:36:58.190018 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260927 18:36:58.192866 17697 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45575
I20260927 18:36:58.193192 17697 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:58.194031 17697 heartbeater.cc:511] Master 127.0.97.126:45575 requested a full tablet report, sending...
I20260927 18:36:58.196261 17508 ts_manager.cc:194] Registered new tserver with Master: f9a0b2ebe67141168247074c95736eb2 (127.0.97.65:38169)
I20260927 18:36:58.197813 17508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:47313
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:36:58.228736 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:58.557703 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:58.558035 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:58.558131 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:58.565266 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:58.565372 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:58.565555 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:36:58.575407 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:58.576678 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:58.578102 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:58.591733 17709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:58.591920 17711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:58.591776 17708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:58.592669 17703 server_base.cc:1034] running on GCE node
I20260927 18:36:58.593228 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:58.594250 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:58.595463 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790534218595398 us; error 57 us; skew 500 ppm
I20260927 18:36:58.595731 17703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:58 dist-test-slave-n3p4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:58.598757 17703 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:36:58.600097 17703 webserver.cc:460] Webserver started at http://127.0.97.66:39845/ using document root <none> and password file <none>
I20260927 18:36:58.600768 17703 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:58.600863 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:58.601121 17703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:58.603904 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "3fd441aeb6f9427c8be2766af099d70e"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "5d760c4fce7c59efb92800d60c6a5526"
server_key_iv: "5a218e1da1a96dc417d1171d24bd011c"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.604601 17703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "3fd441aeb6f9427c8be2766af099d70e"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "5d760c4fce7c59efb92800d60c6a5526"
server_key_iv: "5a218e1da1a96dc417d1171d24bd011c"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.609746 17703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:36:58.613131 17718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.614566 17703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:58.614873 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "3fd441aeb6f9427c8be2766af099d70e"
format_stamp: "Formatted at 2026-09-27 18:36:58 on dist-test-slave-n3p4"
server_key: "5d760c4fce7c59efb92800d60c6a5526"
server_key_iv: "5a218e1da1a96dc417d1171d24bd011c"
server_key_version: "encryptionkey@0"
I20260927 18:36:58.615077 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:58.630954 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:58.635133 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:58.635422 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:58.636317 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:58.637593 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:58.637701 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.637869 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:58.637946 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:58.670996 17703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:42241
I20260927 18:36:58.671051 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:42241 every 8 connection(s)
I20260927 18:36:58.672362 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:58.681702 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17703
I20260927 18:36:58.681951 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:36:58.682513 389 external_mini_cluster.cc:1428] Setting key 775c2665e45673c593022afc26407f0c
Sep 27 18:36:58 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:58.695029 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.674970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52433 (local address 127.0.97.126:45575)
0927 18:36:58.675156 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:58.675161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:58.676339 (+ 1178us) server_negotiation.cc:407] Connection header received
0927 18:36:58.678283 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:58.678288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:58.678357 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:58.678446 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:58.679911 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.680594 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.682244 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.682907 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.683045 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:58.687503 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:58.687512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:58.687520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:58.687564 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:58.691270 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:58.692307 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:58.692316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:58.692322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:58.692480 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:58.692965 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:58.692972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:58.692978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:58.693401 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:58.693573 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:58.694598 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0927 18:36:58.694784 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:36:58.696596 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.675469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45575 (local address 127.0.97.66:52433)
0927 18:36:58.676175 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:58.676212 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:58.678033 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:58.678650 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:58.678657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:58.679170 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:58.679733 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:58.679749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.680735 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.680739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:58.682112 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:58.682123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:58.683126 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:58.683136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:58.683446 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:58.684089 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:58.684150 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:58.687266 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:58.691442 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:58.691449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:58.691462 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:58.692163 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:58.692624 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:58.692629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:58.692631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:58.692838 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:58.693570 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:58.693580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:58.694159 (+ 579us) client_negotiation.cc:769] Sending connection context
0927 18:36:58.694504 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 18:36:58.695115 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260927 18:36:58.698107 17831 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45575
I20260927 18:36:58.698421 17831 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:58.699153 17831 heartbeater.cc:511] Master 127.0.97.126:45575 requested a full tablet report, sending...
I20260927 18:36:58.700618 17508 ts_manager.cc:194] Registered new tserver with Master: 3fd441aeb6f9427c8be2766af099d70e (127.0.97.66:42241)
I20260927 18:36:58.701435 17508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52433
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:36:58.741076 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:59.061724 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:59.062104 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:59.062282 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:59.069346 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:59.069497 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:59.069667 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:36:59.079672 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:36:59.080996 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:59.082361 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:59.095752 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:59.095827 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:59.095818 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:59.096639 17836 server_base.cc:1034] running on GCE node
I20260927 18:36:59.097208 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:59.098096 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:59.099287 17836 hybrid_clock.cc:648] HybridClock initialized: now 1790534219099247 us; error 27 us; skew 500 ppm
I20260927 18:36:59.099587 17836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:36:59 dist-test-slave-n3p4 krb5kdc[17463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:59.102770 17836 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:36:59.103999 17836 webserver.cc:460] Webserver started at http://127.0.97.67:32959/ using document root <none> and password file <none>
I20260927 18:36:59.104610 17836 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:59.104713 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:59.104954 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:59.107738 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "2315a7f29bd143f1a55439a83e9461c2"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-n3p4"
server_key: "37594080ab9a2023c285a148b9676198"
server_key_iv: "6fa105bca8d6b3bbc445d2a4bdc9f6e7"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.108455 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "2315a7f29bd143f1a55439a83e9461c2"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-n3p4"
server_key: "37594080ab9a2023c285a148b9676198"
server_key_iv: "6fa105bca8d6b3bbc445d2a4bdc9f6e7"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.114073 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:36:59.117666 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:59.119071 17836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:59.119254 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "2315a7f29bd143f1a55439a83e9461c2"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-n3p4"
server_key: "37594080ab9a2023c285a148b9676198"
server_key_iv: "6fa105bca8d6b3bbc445d2a4bdc9f6e7"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.119483 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:59.139199 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:59.143327 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:59.143548 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:59.144410 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:59.145601 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:59.145674 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:59.145761 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:59.145865 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:59.179284 17836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45815
I20260927 18:36:59.179486 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45815 every 8 connection(s)
I20260927 18:36:59.180557 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:59.183231 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 17836
I20260927 18:36:59.183405 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:36:59.183938 389 external_mini_cluster.cc:1428] Setting key 1d736aaa81b00a09e8af8b62934d4bb2
Sep 27 18:36:59 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:36:59.200681 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.183071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34623 (local address 127.0.97.126:45575)
0927 18:36:59.183258 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.183263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.185355 (+ 2092us) server_negotiation.cc:407] Connection header received
0927 18:36:59.187313 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.187319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.187407 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.187528 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:59.189130 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.189950 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.191436 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.191837 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.191977 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.195289 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:59.195294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:59.195316 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:59.195359 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:59.197725 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.198461 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.198467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.198471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.198617 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.199025 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.199030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.199035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.199333 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:59.199492 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.200127 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.200350 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:36:59.200821 17697 heartbeater.cc:503] Master 127.0.97.126:45575 was elected leader, sending a full tablet report...
I20260927 18:36:59.201684 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.183725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45575 (local address 127.0.97.67:34623)
0927 18:36:59.185193 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.185243 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.187023 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.187767 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.187776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.188408 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:59.188971 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.188987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.190101 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.190105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.191290 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.191301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.192055 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.192064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.192307 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.192983 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:59.193008 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:59.195090 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:59.197917 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:59.197925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:59.197940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:59.198339 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:59.198735 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:59.198739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:59.198743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:59.198911 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:59.199464 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:59.199471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:59.199785 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.200031 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.200500 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"thread_start_us":975,"threads_started":1}
I20260927 18:36:59.203042 17964 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45575
I20260927 18:36:59.203329 17964 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:59.204028 17964 heartbeater.cc:511] Master 127.0.97.126:45575 requested a full tablet report, sending...
I20260927 18:36:59.205325 17508 ts_manager.cc:194] Registered new tserver with Master: 2315a7f29bd143f1a55439a83e9461c2 (127.0.97.67:45815)
I20260927 18:36:59.206316 17508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:34623
I20260927 18:36:59.211688 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:59.228034 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.216313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.97.126:45575)
0927 18:36:59.216469 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.216473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.216909 (+ 436us) server_negotiation.cc:407] Connection header received
0927 18:36:59.217036 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.217040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.217096 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.217175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:59.218508 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.219263 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.220643 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.221046 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.221144 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.223496 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:59.223501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:59.223504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:59.223531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:59.225938 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.226516 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.226519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.226521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.226631 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.226945 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.226948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.226950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.227239 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:59.227423 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.227683 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.227826 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:36:59.231956 17508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:36:59.238731 17508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:36:59.276496 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.250413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.0.97.65:38169)
0927 18:36:59.265647 (+ 15234us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.265661 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.265699 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:36:59.265847 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.265860 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.266765 (+ 905us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.267009 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:59.268130 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.270162 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.273443 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.274931 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.275055 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.275325 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.275924 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.276092 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15015,"thread_start_us":15931,"threads_started":1}
I20260927 18:36:59.276595 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.250148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38169 (local address 127.0.0.1:36424)
0927 18:36:59.263675 (+ 13527us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.263722 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.263900 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.267370 (+ 3470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.267382 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.267424 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:59.267872 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.267888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.270298 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.270306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.273207 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.273220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.275193 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.275204 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.275449 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.275711 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.275881 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.276093 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13241,"mutex_wait_us":145,"thread_start_us":13095,"threads_started":1}
I20260927 18:36:59.279732 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.250748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:45815 (local address 127.0.0.1:50160)
0927 18:36:59.271380 (+ 20632us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.271399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.271506 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.272248 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.272254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.272279 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:59.272705 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.272719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.274561 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.274565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.277081 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.277095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.278903 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.278908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.279072 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.279102 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.279197 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.279333 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20463,"mutex_wait_us":173,"thread_start_us":20286,"threads_started":1}
I20260927 18:36:59.279786 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.251416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.0.97.67:45815)
0927 18:36:59.270829 (+ 19413us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.270843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.271638 (+ 795us) server_negotiation.cc:407] Connection header received
0927 18:36:59.271756 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.271765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.272030 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.272254 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:59.272953 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.274407 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.277252 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.278705 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.278849 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.279075 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.279258 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.279413 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19219,"thread_start_us":19043,"threads_started":1}
I20260927 18:36:59.280721 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.250531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42241 (local address 127.0.0.1:51690)
0927 18:36:59.269729 (+ 19198us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.269752 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.269905 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.272687 (+ 2782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.272692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.272717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:59.273122 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.273131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.275028 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.275033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.277325 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.277337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.279956 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.279961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.280121 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.280170 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.280266 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.280393 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19042,"mutex_wait_us":1,"thread_start_us":19471,"threads_started":1}
I20260927 18:36:59.281595 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.250929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.0.97.66:42241)
0927 18:36:59.272128 (+ 21199us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.272141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.272169 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:36:59.272268 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.272274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.272483 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.272694 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:59.273276 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.274910 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.277712 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.279512 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.279627 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.279820 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.281118 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.281291 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21037,"thread_start_us":20923,"threads_started":1}
I20260927 18:36:59.281267 17632 tablet_service.cc:1467] Processing CreateTablet for tablet 74ad03625880480dacecbb5551553b2b (DEFAULT_TABLE table=test-table [id=4cd35b98d2b4415ea1e7980920cad29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:59.283560 17899 tablet_service.cc:1467] Processing CreateTablet for tablet 74ad03625880480dacecbb5551553b2b (DEFAULT_TABLE table=test-table [id=4cd35b98d2b4415ea1e7980920cad29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:59.284180 17632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ad03625880480dacecbb5551553b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:59.284883 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ad03625880480dacecbb5551553b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:59.284901 17766 tablet_service.cc:1467] Processing CreateTablet for tablet 74ad03625880480dacecbb5551553b2b (DEFAULT_TABLE table=test-table [id=4cd35b98d2b4415ea1e7980920cad29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:59.286334 17766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ad03625880480dacecbb5551553b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:59.297041 17982 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Bootstrap starting.
I20260927 18:36:59.300041 17983 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Bootstrap starting.
I20260927 18:36:59.301082 17982 tablet_bootstrap.cc:654] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:59.301678 17984 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Bootstrap starting.
I20260927 18:36:59.302484 17982 log.cc:824] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:59.304615 17983 tablet_bootstrap.cc:654] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:59.304922 17984 tablet_bootstrap.cc:654] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:59.305760 17982 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: No bootstrap required, opened a new log
I20260927 18:36:59.305861 17983 log.cc:824] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:59.305936 17984 log.cc:824] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:59.306097 17982 ts_tablet_manager.cc:1397] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260927 18:36:59.309090 17983 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: No bootstrap required, opened a new log
I20260927 18:36:59.309432 17983 ts_tablet_manager.cc:1397] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 18:36:59.315014 17984 tablet_bootstrap.cc:492] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: No bootstrap required, opened a new log
I20260927 18:36:59.315297 17984 ts_tablet_manager.cc:1397] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 18:36:59.316624 17982 raft_consensus.cc:348] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.317077 17982 raft_consensus.cc:374] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:59.317184 17982 raft_consensus.cc:729] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2315a7f29bd143f1a55439a83e9461c2, State: Initialized, Role: FOLLOWER
I20260927 18:36:59.317909 17982 consensus_queue.cc:260] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.319036 17964 heartbeater.cc:503] Master 127.0.97.126:45575 was elected leader, sending a full tablet report...
I20260927 18:36:59.319811 17982 ts_tablet_manager.cc:1428] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260927 18:36:59.320474 17983 raft_consensus.cc:348] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.322088 17983 raft_consensus.cc:374] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:59.322211 17983 raft_consensus.cc:729] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a0b2ebe67141168247074c95736eb2, State: Initialized, Role: FOLLOWER
I20260927 18:36:59.322969 17983 consensus_queue.cc:260] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.324827 17983 ts_tablet_manager.cc:1428] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260927 18:36:59.326584 17984 raft_consensus.cc:348] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.326989 17984 raft_consensus.cc:374] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:59.327088 17984 raft_consensus.cc:729] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd441aeb6f9427c8be2766af099d70e, State: Initialized, Role: FOLLOWER
I20260927 18:36:59.327708 17984 consensus_queue.cc:260] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.328859 17831 heartbeater.cc:503] Master 127.0.97.126:45575 was elected leader, sending a full tablet report...
I20260927 18:36:59.329448 17984 ts_tablet_manager.cc:1428] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260927 18:36:59.350342 17990 raft_consensus.cc:482] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:59.350620 17990 raft_consensus.cc:504] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.352299 17990 leader_election.cc:288] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a0b2ebe67141168247074c95736eb2 (127.0.97.65:38169), 2315a7f29bd143f1a55439a83e9461c2 (127.0.97.67:45815)
I20260927 18:36:59.362205 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.352636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35079 (local address 127.0.97.65:38169)
0927 18:36:59.352853 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.352860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.353827 (+ 967us) server_negotiation.cc:407] Connection header received
0927 18:36:59.354190 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.354198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.354277 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.354363 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:59.355300 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.356611 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.359884 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.361264 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.361373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.361452 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.361766 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.361918 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:36:59.362282 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.352743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38169 (local address 127.0.97.66:35079)
0927 18:36:59.353655 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.353677 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.353964 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.354531 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.354537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.354560 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:59.355104 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.355113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.356783 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.356790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.359450 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.359469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.361464 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.361468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.361590 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.361624 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.361739 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.361891 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":188,"thread_start_us":475,"threads_started":1}
I20260927 18:36:59.363355 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.352748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:45815 (local address 127.0.97.66:51215)
0927 18:36:59.353775 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:59.353830 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:59.354026 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:59.354549 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:59.354552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:59.354568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:59.355056 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.355071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.356740 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.356747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.359469 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:59.359478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.362814 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.362818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:59.362930 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.362951 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:36:59.363024 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:36:59.363105 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":196,"thread_start_us":811,"threads_started":1}
I20260927 18:36:59.363233 17652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ad03625880480dacecbb5551553b2b" candidate_uuid: "3fd441aeb6f9427c8be2766af099d70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a0b2ebe67141168247074c95736eb2" is_pre_election: true
I20260927 18:36:59.363481 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.352799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51215 (local address 127.0.97.67:45815)
0927 18:36:59.353026 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.353034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.353916 (+ 882us) server_negotiation.cc:407] Connection header received
0927 18:36:59.354245 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.354251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.354327 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.354389 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:59.355262 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.356577 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.361520 (+ 4943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.362483 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.362594 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.362698 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.363081 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.363228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:36:59.363848 17652 raft_consensus.cc:2454] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd441aeb6f9427c8be2766af099d70e in term 0.
I20260927 18:36:59.364529 17719 leader_election.cc:302] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd441aeb6f9427c8be2766af099d70e, f9a0b2ebe67141168247074c95736eb2; no voters:
I20260927 18:36:59.364933 17990 raft_consensus.cc:2781] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:59.365051 17990 raft_consensus.cc:482] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:59.365167 17990 raft_consensus.cc:3037] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:59.365163 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ad03625880480dacecbb5551553b2b" candidate_uuid: "3fd441aeb6f9427c8be2766af099d70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2315a7f29bd143f1a55439a83e9461c2" is_pre_election: true
I20260927 18:36:59.365677 17919 raft_consensus.cc:2454] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd441aeb6f9427c8be2766af099d70e in term 0.
I20260927 18:36:59.367669 17990 raft_consensus.cc:504] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.368328 17990 leader_election.cc:288] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [CANDIDATE]: Term 1 election: Requested vote from peers f9a0b2ebe67141168247074c95736eb2 (127.0.97.65:38169), 2315a7f29bd143f1a55439a83e9461c2 (127.0.97.67:45815)
I20260927 18:36:59.368707 17652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ad03625880480dacecbb5551553b2b" candidate_uuid: "3fd441aeb6f9427c8be2766af099d70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a0b2ebe67141168247074c95736eb2"
I20260927 18:36:59.368920 17652 raft_consensus.cc:3037] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:59.368955 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ad03625880480dacecbb5551553b2b" candidate_uuid: "3fd441aeb6f9427c8be2766af099d70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2315a7f29bd143f1a55439a83e9461c2"
I20260927 18:36:59.369119 17919 raft_consensus.cc:3037] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:59.372146 17652 raft_consensus.cc:2454] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd441aeb6f9427c8be2766af099d70e in term 1.
I20260927 18:36:59.372326 17919 raft_consensus.cc:2454] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd441aeb6f9427c8be2766af099d70e in term 1.
I20260927 18:36:59.372605 17719 leader_election.cc:302] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd441aeb6f9427c8be2766af099d70e, f9a0b2ebe67141168247074c95736eb2; no voters:
I20260927 18:36:59.372973 17990 raft_consensus.cc:2781] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:59.373425 17990 raft_consensus.cc:686] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 1 LEADER]: Becoming Leader. State: Replica: 3fd441aeb6f9427c8be2766af099d70e, State: Running, Role: LEADER
I20260927 18:36:59.373929 17990 consensus_queue.cc:237] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } }
I20260927 18:36:59.378278 17508 catalog_manager.cc:5495] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fd441aeb6f9427c8be2766af099d70e (127.0.97.66). New cstate: current_term: 1 leader_uuid: "3fd441aeb6f9427c8be2766af099d70e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd441aeb6f9427c8be2766af099d70e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 } health_report { overall_health: UNKNOWN } } }
I20260927 18:36:59.406098 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.400409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.0.97.66:42241)
0927 18:36:59.400691 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.400696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.400714 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:36:59.400878 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.400882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.400956 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.401073 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:59.401414 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.402223 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.403741 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.404224 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.404339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.404830 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:59.405392 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:59.405537 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.405588 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.405668 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:36:59.414443 17919 raft_consensus.cc:1261] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 1 FOLLOWER]: Refusing update from remote peer 3fd441aeb6f9427c8be2766af099d70e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:59.414788 17652 raft_consensus.cc:1261] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 1 FOLLOWER]: Refusing update from remote peer 3fd441aeb6f9427c8be2766af099d70e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:59.415419 17993 consensus_queue.cc:1035] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2315a7f29bd143f1a55439a83e9461c2" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:59.415864 17990 consensus_queue.cc:1035] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a0b2ebe67141168247074c95736eb2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:36:59.422091 17698 tablet.cc:2369] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:36:59.424343 17832 tablet.cc:2369] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:59.443485 17508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:59.443759 17508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:59.447351 17508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ddc59a34f294530be981409c4676621: Sending DeleteTablet for 3 replicas of tablet 74ad03625880480dacecbb5551553b2b
I20260927 18:36:59.448434 17766 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ad03625880480dacecbb5551553b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51690
I20260927 18:36:59.449119 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ad03625880480dacecbb5551553b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50160
I20260927 18:36:59.449151 17632 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ad03625880480dacecbb5551553b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36424
I20260927 18:36:59.450330 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17569
I20260927 18:36:59.473366 18008 tablet_replica.cc:330] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: stopping tablet replica
I20260927 18:36:59.474880 18008 raft_consensus.cc:2229] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:59.475596 18008 raft_consensus.cc:2258] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:59.476271 18006 tablet_replica.cc:330] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: stopping tablet replica
I20260927 18:36:59.477080 18006 raft_consensus.cc:2229] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:36:59.478084 18006 raft_consensus.cc:2258] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:59.480074 18008 ts_tablet_manager.cc:1905] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:59.480363 18006 ts_tablet_manager.cc:1905] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:36:59 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:36:59.484905 18007 tablet_replica.cc:330] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: stopping tablet replica
I20260927 18:36:59.485347 18007 raft_consensus.cc:2229] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:59.485972 18007 raft_consensus.cc:2258] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:59.490056 18007 ts_tablet_manager.cc:1905] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:59.490509 18008 ts_tablet_manager.cc:1918] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:59.490748 18008 log.cc:1196] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/74ad03625880480dacecbb5551553b2b
I20260927 18:36:59.491411 18008 ts_tablet_manager.cc:1939] T 74ad03625880480dacecbb5551553b2b P f9a0b2ebe67141168247074c95736eb2: Deleting consensus metadata
I20260927 18:36:59.493538 17492 catalog_manager.cc:4841] TS f9a0b2ebe67141168247074c95736eb2 (127.0.97.65:38169): tablet 74ad03625880480dacecbb5551553b2b (table test-table [id=4cd35b98d2b4415ea1e7980920cad29a]) successfully deleted
I20260927 18:36:59.494804 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.451288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.0.97.65:38169)
0927 18:36:59.451478 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.451484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.474221 (+ 22737us) server_negotiation.cc:407] Connection header received
0927 18:36:59.474445 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.474451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.474557 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.474660 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:59.476615 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.477769 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.479810 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.480446 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.480541 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.486125 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:59.486140 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:59.486157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:59.486199 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:59.491192 (+ 4993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.492080 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.492090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.492093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.492266 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.492753 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.492758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.492762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.493224 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:59.493505 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.494174 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.494398 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:36:59.496258 17692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:59.496995 18007 ts_tablet_manager.cc:1918] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:59.497195 18007 log.cc:1196] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/74ad03625880480dacecbb5551553b2b
I20260927 18:36:59.501951 18006 ts_tablet_manager.cc:1918] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:59.502226 18006 log.cc:1196] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/74ad03625880480dacecbb5551553b2b
I20260927 18:36:59.502743 18006 ts_tablet_manager.cc:1939] T 74ad03625880480dacecbb5551553b2b P 3fd441aeb6f9427c8be2766af099d70e: Deleting consensus metadata
I20260927 18:36:59.504252 18007 ts_tablet_manager.cc:1939] T 74ad03625880480dacecbb5551553b2b P 2315a7f29bd143f1a55439a83e9461c2: Deleting consensus metadata
I20260927 18:36:59.504371 17493 catalog_manager.cc:4841] TS 3fd441aeb6f9427c8be2766af099d70e (127.0.97.66:42241): tablet 74ad03625880480dacecbb5551553b2b (table test-table [id=4cd35b98d2b4415ea1e7980920cad29a]) successfully deleted
I20260927 18:36:59.505925 17494 catalog_manager.cc:4841] TS 2315a7f29bd143f1a55439a83e9461c2 (127.0.97.67:45815): tablet 74ad03625880480dacecbb5551553b2b (table test-table [id=4cd35b98d2b4415ea1e7980920cad29a]) successfully deleted
I20260927 18:36:59.665548 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17569
I20260927 18:36:59.687776 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17703
Sep 27 18:36:59 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:36:59.708815 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.689170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.0.97.66:42241)
0927 18:36:59.689376 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.689382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.689407 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:36:59.689478 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.689483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.689561 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.689684 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:59.691474 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.692752 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.694596 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.695297 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.695422 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.700821 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:59.700835 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:59.700858 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:59.700909 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:59.705506 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.706502 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.706516 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.706520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.706710 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.707201 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.707206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.707210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.707638 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:59.708011 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.708274 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.708442 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:36:59.710273 17826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:59.827517 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17703
I20260927 18:36:59.845774 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17836
Sep 27 18:36:59 dist-test-slave-n3p4 krb5kdc[17463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:36:59.860899 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:59.846571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.97.67:45815)
0927 18:36:59.846754 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:59.846761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:59.846862 (+ 101us) server_negotiation.cc:407] Connection header received
0927 18:36:59.847082 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:59.847086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:59.847152 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:59.847264 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:59.848581 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.849307 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.850684 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:59.851072 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:59.851213 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:59.855801 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:59.855815 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:59.855831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:59.855867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:59.858469 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.859139 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.859146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.859148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.859294 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:59.859643 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:59.859646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:59.859648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:59.859964 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:59.860185 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:59.860483 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:36:59.860643 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:36:59.861505 17959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:36:59.951169 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17836
I20260927 18:36:59.967679 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 17476
I20260927 18:36:59.968743 17538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:00.079296 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 17476
2026-09-27T18:37:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:00Z Disabled control of system clock
I20260927 18:37:00.154482 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38411
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:00.481539 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:00.481853 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:00.482013 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:00.489475 18017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:00.489609 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:00.489681 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:00.489761 18017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:00.489848 18017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:00.499087 18017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:00.500398 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:00.502318 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:00.514048 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.514132 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.514019 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:00.514822 18017 server_base.cc:1034] running on GCE node
I20260927 18:37:00.515618 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:00.516950 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:00.518152 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790534220518088 us; error 54 us; skew 500 ppm
I20260927 18:37:00.518492 18017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:00.524979 18017 webserver.cc:460] Webserver started at http://127.0.97.126:35909/ using document root <none> and password file <none>
I20260927 18:37:00.525678 18017 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:00.525810 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:00.526151 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:00.528991 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "0c5971a6cbaa49258617b4a1f43055f6"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-n3p4"
server_key: "798b1e2cdcb1f33b5403490babcb7f28"
server_key_iv: "2f7b64d06c78f164e2904a7183ccf5b1"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.529845 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "0c5971a6cbaa49258617b4a1f43055f6"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-n3p4"
server_key: "798b1e2cdcb1f33b5403490babcb7f28"
server_key_iv: "2f7b64d06c78f164e2904a7183ccf5b1"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.535039 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:37:00.538668 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.540073 18017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:00.540274 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "0c5971a6cbaa49258617b4a1f43055f6"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-n3p4"
server_key: "798b1e2cdcb1f33b5403490babcb7f28"
server_key_iv: "2f7b64d06c78f164e2904a7183ccf5b1"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.540489 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:00.557868 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:00.558691 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:00.558966 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:00.577498 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38411
I20260927 18:37:00.577521 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38411 every 8 connection(s)
I20260927 18:37:00.578898 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:00.582664 18083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:00.586787 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18017
I20260927 18:37:00.586970 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:00.587527 389 external_mini_cluster.cc:1428] Setting key 53a13406f69bd9117e29632181e15502
I20260927 18:37:00.594158 18083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: Bootstrap starting.
I20260927 18:37:00.597658 18083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:00.598860 18083 log.cc:824] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:00.602277 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.589426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.0.97.126:38411)
0927 18:37:00.590422 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.590464 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.590516 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:37:00.592712 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.592734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.593195 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.593496 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.594161 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.595555 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.597441 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.598243 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.598370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.599739 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.599754 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.599778 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:00.600514 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.600596 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:00.600651 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.600739 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.600982 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.601274 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":396,"threads_started":1}
I20260927 18:37:00.602360 18083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: No bootstrap required, opened a new log
I20260927 18:37:00.610026 18083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } }
I20260927 18:37:00.610399 18083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:00.610534 18083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c5971a6cbaa49258617b4a1f43055f6, State: Initialized, Role: FOLLOWER
I20260927 18:37:00.611097 18083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } }
I20260927 18:37:00.611344 18083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:00.611481 18083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:00.611691 18083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:00.613996 18083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } }
I20260927 18:37:00.614506 18083 leader_election.cc:302] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c5971a6cbaa49258617b4a1f43055f6; no voters:
I20260927 18:37:00.615097 18083 leader_election.cc:288] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:00.615281 18088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:00.616039 18088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [term 1 LEADER]: Becoming Leader. State: Replica: 0c5971a6cbaa49258617b4a1f43055f6, State: Running, Role: LEADER
I20260927 18:37:00.616519 18088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } }
I20260927 18:37:00.617312 18083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:00.619361 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c5971a6cbaa49258617b4a1f43055f6. Latest consensus state: current_term: 1 leader_uuid: "0c5971a6cbaa49258617b4a1f43055f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } } }
I20260927 18:37:00.619870 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:00.621969 18089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c5971a6cbaa49258617b4a1f43055f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5971a6cbaa49258617b4a1f43055f6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38411 } } }
I20260927 18:37:00.622229 18089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:00.622745 18094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:00.627972 18094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:00.636554 18094 catalog_manager.cc:1347] Generated new cluster ID: 9b537748d53f4341a8fb37b0a5f4058c
I20260927 18:37:00.636685 18094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:00.654670 18094 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:00.655889 18094 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:00.667192 18094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: Generated new TSK 0
I20260927 18:37:00.668133 18094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:00.683894 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--builtin_ntp_servers=127.0.97.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:01.019749 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:01.020059 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:01.020210 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:01.027678 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:01.027817 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:01.028008 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:37:01.037002 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:01.038340 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:01.040150 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:01.054068 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.054091 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.054409 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:01.055241 18107 server_base.cc:1034] running on GCE node
I20260927 18:37:01.055909 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:01.056922 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:01.058146 18107 hybrid_clock.cc:648] HybridClock initialized: now 1790534221058088 us; error 33 us; skew 500 ppm
I20260927 18:37:01.058530 18107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:01.060931 18107 webserver.cc:460] Webserver started at http://127.0.97.65:32977/ using document root <none> and password file <none>
I20260927 18:37:01.061667 18107 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:01.061827 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:01.062242 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:01.065191 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "0ad0955309284c17a3fa197974176e69"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "f390434e23aded22c28851f7696c3edc"
server_key_iv: "e22137e8824fa6ba9516f061280f49d0"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.066077 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0ad0955309284c17a3fa197974176e69"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "f390434e23aded22c28851f7696c3edc"
server_key_iv: "e22137e8824fa6ba9516f061280f49d0"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.071323 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 18:37:01.074644 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.075950 18107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:37:01.076118 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "0ad0955309284c17a3fa197974176e69"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "f390434e23aded22c28851f7696c3edc"
server_key_iv: "e22137e8824fa6ba9516f061280f49d0"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.076332 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:01.095775 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:01.096578 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:01.096856 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:01.097891 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:01.099313 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:01.099442 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.099571 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:01.099661 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.133168 18107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39721
I20260927 18:37:01.133262 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39721 every 8 connection(s)
I20260927 18:37:01.134589 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:01.140254 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18107
I20260927 18:37:01.140416 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:01.140797 389 external_mini_cluster.cc:1428] Setting key d9ba69640987c708e8a27bdd434614f6
I20260927 18:37:01.143458 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--builtin_ntp_servers=127.0.97.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:01.147511 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:01.137077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40111 (local address 127.0.97.126:38411)
0927 18:37:01.137530 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:01.137536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:01.138766 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 18:37:01.140510 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:01.140516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:01.140608 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:01.140727 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:01.141965 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.142856 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.144622 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.145098 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.145226 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:01.146653 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:01.146657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:01.146661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:01.146677 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:01.146718 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:01.146730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:01.146834 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:01.147065 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 18:37:01.147159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260927 18:37:01.148470 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:01.137698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38411 (local address 127.0.97.65:40111)
0927 18:37:01.138606 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:01.138646 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:01.140210 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:01.140982 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:01.140991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:01.141075 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:01.141746 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:01.141763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.143010 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.143015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:01.144463 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:01.144475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.145341 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.145348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:01.145570 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:01.146330 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:01.146333 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:01.146384 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:01.146464 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:01.146832 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:01.146844 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:01.146855 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:37:01.147034 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 18:37:01.147432 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":249,"threads_started":1}
I20260927 18:37:01.149956 18234 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38411
I20260927 18:37:01.150282 18234 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:01.151011 18234 heartbeater.cc:511] Master 127.0.97.126:38411 requested a full tablet report, sending...
I20260927 18:37:01.153088 18048 ts_manager.cc:194] Registered new tserver with Master: 0ad0955309284c17a3fa197974176e69 (127.0.97.65:39721)
I20260927 18:37:01.154587 18048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:40111
W20260927 18:37:01.473809 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:01.474258 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:01.474472 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:01.482990 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:01.483081 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:01.483249 18238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:37:01.492272 18238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:01.493564 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:01.495378 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:01.508257 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.508594 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.508307 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:01.509191 18238 server_base.cc:1034] running on GCE node
I20260927 18:37:01.509847 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:01.510860 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:01.512135 18238 hybrid_clock.cc:648] HybridClock initialized: now 1790534221512077 us; error 79 us; skew 500 ppm
I20260927 18:37:01.512499 18238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:01.515007 18238 webserver.cc:460] Webserver started at http://127.0.97.66:44313/ using document root <none> and password file <none>
I20260927 18:37:01.515720 18238 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:01.515854 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:01.516175 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:01.519407 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "40a7159299af4f4b9cb9d4a89e96813a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "bc3fc8bd4bb8194aac4cc53645a28062"
server_key_iv: "2cf8829026c06821e6c9d0f4fd41315e"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.520231 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "40a7159299af4f4b9cb9d4a89e96813a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "bc3fc8bd4bb8194aac4cc53645a28062"
server_key_iv: "2cf8829026c06821e6c9d0f4fd41315e"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.525197 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:37:01.528604 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.530023 18238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:01.530205 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "40a7159299af4f4b9cb9d4a89e96813a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "bc3fc8bd4bb8194aac4cc53645a28062"
server_key_iv: "2cf8829026c06821e6c9d0f4fd41315e"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.530445 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:01.542703 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:01.543485 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:01.543751 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:01.544751 18238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:01.546259 18238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:01.546391 18238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.546530 18238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:01.546622 18238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.579892 18238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39249
I20260927 18:37:01.579959 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39249 every 8 connection(s)
I20260927 18:37:01.581259 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:01.589378 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18238
I20260927 18:37:01.589599 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:01.590243 389 external_mini_cluster.cc:1428] Setting key 9615e297619233608666ef1c6f88aa48
I20260927 18:37:01.593365 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--builtin_ntp_servers=127.0.97.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:01.595636 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:01.584063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57059 (local address 127.0.97.126:38411)
0927 18:37:01.584252 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:01.584257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:01.585853 (+ 1596us) server_negotiation.cc:407] Connection header received
0927 18:37:01.587722 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:01.587728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:01.587826 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:01.587937 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:01.589005 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.589940 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.591842 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.592553 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.592785 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:01.594636 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:01.594645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:01.594656 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:01.594682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:01.594728 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:01.594745 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:01.594845 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:01.595106 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:37:01.595226 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:37:01.596635 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:01.584710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38411 (local address 127.0.97.66:57059)
0927 18:37:01.585655 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:01.585709 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:01.587450 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:01.588168 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:01.588178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:01.588271 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:01.588833 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:01.588849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.590087 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.590092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:01.591692 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:01.591705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:01.593054 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:01.593062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:01.593381 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:01.594250 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:01.594255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:01.594315 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:01.594419 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:01.594860 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:01.594871 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:01.594880 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:37:01.595084 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:37:01.595487 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":313,"threads_started":1}
I20260927 18:37:01.598161 18365 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38411
I20260927 18:37:01.598487 18365 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:01.599275 18365 heartbeater.cc:511] Master 127.0.97.126:38411 requested a full tablet report, sending...
I20260927 18:37:01.600857 18048 ts_manager.cc:194] Registered new tserver with Master: 40a7159299af4f4b9cb9d4a89e96813a (127.0.97.66:39249)
I20260927 18:37:01.601718 18048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:57059
W20260927 18:37:01.932366 18368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:01.932719 18368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:01.932817 18368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:01.940524 18368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:01.940647 18368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:01.940766 18368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:37:01.950099 18368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:01.951390 18368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:01.953157 18368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:01.966919 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.967042 18376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:01.967198 18374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:01.968783 18368 server_base.cc:1034] running on GCE node
I20260927 18:37:01.969357 18368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:01.970391 18368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:01.971616 18368 hybrid_clock.cc:648] HybridClock initialized: now 1790534221971555 us; error 52 us; skew 500 ppm
I20260927 18:37:01.971931 18368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:01.974237 18368 webserver.cc:460] Webserver started at http://127.0.97.67:45383/ using document root <none> and password file <none>
I20260927 18:37:01.974921 18368 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:01.975052 18368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:01.975363 18368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:01.978176 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "14f9ab5a180d45b5862663e07fdc033a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "d312106ba69d8eeea2ecad1321c53767"
server_key_iv: "56e877e9feb792699021e5b485b22bc8"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.978945 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "14f9ab5a180d45b5862663e07fdc033a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "d312106ba69d8eeea2ecad1321c53767"
server_key_iv: "56e877e9feb792699021e5b485b22bc8"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.983892 18368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:37:01.987198 18382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:01.988432 18368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:37:01.988602 18368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "14f9ab5a180d45b5862663e07fdc033a"
format_stamp: "Formatted at 2026-09-27 18:37:01 on dist-test-slave-n3p4"
server_key: "d312106ba69d8eeea2ecad1321c53767"
server_key_iv: "56e877e9feb792699021e5b485b22bc8"
server_key_version: "encryptionkey@0"
I20260927 18:37:01.988826 18368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:02.007140 18368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:02.007933 18368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:02.008212 18368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:02.009166 18368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:02.010624 18368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:02.010754 18368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:02.010888 18368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:02.010985 18368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:02.045315 18368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33499
I20260927 18:37:02.045369 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33499 every 8 connection(s)
I20260927 18:37:02.046743 18368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:02.048417 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18368
I20260927 18:37:02.048604 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:02.049063 389 external_mini_cluster.cc:1428] Setting key f9383a418cb7a4c488c687390bef1d4d
I20260927 18:37:02.059535 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.049404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43249 (local address 127.0.97.126:38411)
0927 18:37:02.049601 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.049607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.051431 (+ 1824us) server_negotiation.cc:407] Connection header received
0927 18:37:02.053346 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.053350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.053421 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.053507 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:02.054667 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.055376 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.056951 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.057364 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.057465 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.058904 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:02.058908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:02.058912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:02.058929 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:02.058960 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:02.058971 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:02.059043 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.059231 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.059305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:02.060477 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.050100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38411 (local address 127.0.97.67:43249)
0927 18:37:02.051266 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.051317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.053103 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.053710 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.053741 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.053910 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:02.054462 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.054478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.055506 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.055509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.056787 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.056798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.057546 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.057550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.057757 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.058591 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:02.058594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:02.058632 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:02.058706 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:02.059056 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:02.059062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:02.059068 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.059191 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.059462 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":325,"threads_started":1}
I20260927 18:37:02.061836 18495 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38411
I20260927 18:37:02.062172 18495 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:02.062925 18495 heartbeater.cc:511] Master 127.0.97.126:38411 requested a full tablet report, sending...
I20260927 18:37:02.064466 18048 ts_manager.cc:194] Registered new tserver with Master: 14f9ab5a180d45b5862663e07fdc033a (127.0.97.67:33499)
I20260927 18:37:02.065235 18048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:43249
I20260927 18:37:02.076295 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:02.086465 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.080492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.0.97.126:38411)
0927 18:37:02.080693 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.080697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.081133 (+ 436us) server_negotiation.cc:407] Connection header received
0927 18:37:02.081331 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.081335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.081408 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.081487 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:02.082003 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.082861 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.084192 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.084647 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.084759 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.085854 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:02.085862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:02.085865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:02.085884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:02.085914 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:02.085929 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:02.086007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.086154 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.086228 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:37:02.090171 18048 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:02.097198 18048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:02.144824 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.108909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44572)
0927 18:37:02.134667 (+ 25758us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.134684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.134881 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.136035 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.136043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.136073 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:02.136440 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.136454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.138570 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.138575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.140850 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.140865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.143766 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.143772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.143987 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.144039 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.144155 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.144342 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25627,"thread_start_us":31810,"threads_started":1}
I20260927 18:37:02.145112 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.109863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.0.97.65:39721)
0927 18:37:02.132255 (+ 22392us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.132271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.134780 (+ 2509us) server_negotiation.cc:407] Connection header received
0927 18:37:02.135097 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.135106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.135475 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.135660 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:02.136636 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.138394 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.142146 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.143571 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.143737 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.144440 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.144525 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.144691 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22116,"thread_start_us":22420,"threads_started":1}
I20260927 18:37:02.149359 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.109319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33499 (local address 127.0.0.1:38686)
0927 18:37:02.141453 (+ 32134us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.141471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.141589 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.142285 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.142289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.142318 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:02.142712 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.142722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.144707 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.144712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.147174 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.147190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.148506 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.148512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.148674 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.148704 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.148791 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.148922 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31988,"mutex_wait_us":54,"thread_start_us":24206,"threads_started":1}
I20260927 18:37:02.149354 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.109598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.0.97.67:33499)
0927 18:37:02.139491 (+ 29893us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.139512 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.141612 (+ 2100us) server_negotiation.cc:407] Connection header received
0927 18:37:02.141765 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.141797 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.142093 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.142287 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:02.142932 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.144542 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.147379 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.148297 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.148473 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.148769 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.148842 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.148995 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29656,"thread_start_us":29845,"threads_started":1}
I20260927 18:37:02.149736 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.108764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39249 (local address 127.0.0.1:47014)
0927 18:37:02.133051 (+ 24287us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.133097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.133327 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.142684 (+ 9357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.142693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.142716 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:02.143104 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.143116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.144937 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.144945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.147091 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.147108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.148970 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.148977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.149144 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.149178 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.149262 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.149394 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24015,"mutex_wait_us":74,"thread_start_us":32371,"threads_started":1}
I20260927 18:37:02.150053 18169 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 (DEFAULT_TABLE table=test-table [id=c96758ee5ab54dc494b3438ca0c8b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:02.150988 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.109115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.97.66:39249)
0927 18:37:02.142095 (+ 32980us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.142106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.142135 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:37:02.142251 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.142257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.142473 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.142668 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:02.143268 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.144755 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.147353 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.148747 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.149710 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.150411 (+ 701us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.150473 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.150629 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32828,"thread_start_us":32689,"threads_started":1}
I20260927 18:37:02.151636 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9bea116d64c4ab03e20ad33cbf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:02.153568 18430 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 (DEFAULT_TABLE table=test-table [id=c96758ee5ab54dc494b3438ca0c8b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:02.153959 18300 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 (DEFAULT_TABLE table=test-table [id=c96758ee5ab54dc494b3438ca0c8b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:02.154937 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9bea116d64c4ab03e20ad33cbf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:02.155035 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9bea116d64c4ab03e20ad33cbf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:02.158136 18234 heartbeater.cc:503] Master 127.0.97.126:38411 was elected leader, sending a full tablet report...
I20260927 18:37:02.166445 18513 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Bootstrap starting.
I20260927 18:37:02.170204 18513 tablet_bootstrap.cc:654] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:02.170542 18514 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Bootstrap starting.
I20260927 18:37:02.171628 18513 log.cc:824] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:02.172179 18515 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Bootstrap starting.
I20260927 18:37:02.174258 18514 tablet_bootstrap.cc:654] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:02.175511 18514 log.cc:824] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:02.175596 18515 tablet_bootstrap.cc:654] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:02.175750 18513 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: No bootstrap required, opened a new log
I20260927 18:37:02.176024 18513 ts_tablet_manager.cc:1397] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 18:37:02.176870 18515 log.cc:824] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:02.178483 18514 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: No bootstrap required, opened a new log
I20260927 18:37:02.178784 18514 ts_tablet_manager.cc:1397] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 18:37:02.179602 18515 tablet_bootstrap.cc:492] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: No bootstrap required, opened a new log
I20260927 18:37:02.179853 18515 ts_tablet_manager.cc:1397] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:37:02.186455 18513 raft_consensus.cc:348] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.186918 18513 raft_consensus.cc:374] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:02.187055 18513 raft_consensus.cc:729] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a7159299af4f4b9cb9d4a89e96813a, State: Initialized, Role: FOLLOWER
I20260927 18:37:02.187224 18515 raft_consensus.cc:348] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.187646 18515 raft_consensus.cc:374] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:02.187752 18515 raft_consensus.cc:729] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9ab5a180d45b5862663e07fdc033a, State: Initialized, Role: FOLLOWER
I20260927 18:37:02.187707 18513 consensus_queue.cc:260] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.188452 18515 consensus_queue.cc:260] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.188848 18365 heartbeater.cc:503] Master 127.0.97.126:38411 was elected leader, sending a full tablet report...
I20260927 18:37:02.189582 18495 heartbeater.cc:503] Master 127.0.97.126:38411 was elected leader, sending a full tablet report...
I20260927 18:37:02.190111 18515 ts_tablet_manager.cc:1428] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260927 18:37:02.189198 18514 raft_consensus.cc:348] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.190697 18514 raft_consensus.cc:374] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:02.190814 18514 raft_consensus.cc:729] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad0955309284c17a3fa197974176e69, State: Initialized, Role: FOLLOWER
I20260927 18:37:02.191349 18514 consensus_queue.cc:260] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.192103 18513 ts_tablet_manager.cc:1428] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260927 18:37:02.193218 18514 ts_tablet_manager.cc:1428] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
W20260927 18:37:02.300073 18496 tablet.cc:2369] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:02.333515 18366 tablet.cc:2369] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:02.386965 18235 tablet.cc:2369] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:02.404464 18520 raft_consensus.cc:482] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:02.404771 18520 raft_consensus.cc:504] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.406487 18520 leader_election.cc:288] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a7159299af4f4b9cb9d4a89e96813a (127.0.97.66:39249), 0ad0955309284c17a3fa197974176e69 (127.0.97.65:39721)
I20260927 18:37:02.413024 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.406677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39249 (local address 127.0.97.67:59609)
0927 18:37:02.406846 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.406871 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.407025 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.407541 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.407557 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.407589 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:02.407841 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.407857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.409156 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.409160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.411137 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.411151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.412472 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.412476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.412586 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.412610 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.412682 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.412795 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":5}
I20260927 18:37:02.413132 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.406845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59609 (local address 127.0.97.66:39249)
0927 18:37:02.407049 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.407055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.407078 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:02.407184 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.407189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.407288 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.407400 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:02.408046 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.409015 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.411313 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.412330 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.412459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.412552 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.412733 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.412863 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:02.413905 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.407188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56443 (local address 127.0.97.65:39721)
0927 18:37:02.407429 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.407438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.407981 (+ 543us) server_negotiation.cc:407] Connection header received
0927 18:37:02.408061 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.408066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.408159 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.408266 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:02.408759 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.409588 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.411804 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.412996 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.413124 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.413249 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.413515 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.413658 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:37:02.413970 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.407040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.97.67:56443)
0927 18:37:02.407815 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.407833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:02.407961 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:02.408289 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:02.408292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:02.408311 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:02.408591 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.408597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.409746 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.409751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.411609 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:02.411619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.413178 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.413185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:02.413348 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.413379 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:37:02.413468 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:37:02.413584 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":50,"thread_start_us":410,"threads_started":1}
I20260927 18:37:02.414522 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9bea116d64c4ab03e20ad33cbf446" candidate_uuid: "14f9ab5a180d45b5862663e07fdc033a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a7159299af4f4b9cb9d4a89e96813a" is_pre_election: true
I20260927 18:37:02.415055 18320 raft_consensus.cc:2454] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f9ab5a180d45b5862663e07fdc033a in term 0.
I20260927 18:37:02.415002 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9bea116d64c4ab03e20ad33cbf446" candidate_uuid: "14f9ab5a180d45b5862663e07fdc033a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad0955309284c17a3fa197974176e69" is_pre_election: true
I20260927 18:37:02.415544 18189 raft_consensus.cc:2454] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f9ab5a180d45b5862663e07fdc033a in term 0.
I20260927 18:37:02.415678 18383 leader_election.cc:302] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f9ab5a180d45b5862663e07fdc033a, 40a7159299af4f4b9cb9d4a89e96813a; no voters:
I20260927 18:37:02.416083 18520 raft_consensus.cc:2781] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:02.416193 18520 raft_consensus.cc:482] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:02.416337 18520 raft_consensus.cc:3037] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:02.418666 18520 raft_consensus.cc:504] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.419355 18520 leader_election.cc:288] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [CANDIDATE]: Term 1 election: Requested vote from peers 40a7159299af4f4b9cb9d4a89e96813a (127.0.97.66:39249), 0ad0955309284c17a3fa197974176e69 (127.0.97.65:39721)
I20260927 18:37:02.419763 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9bea116d64c4ab03e20ad33cbf446" candidate_uuid: "14f9ab5a180d45b5862663e07fdc033a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a7159299af4f4b9cb9d4a89e96813a"
I20260927 18:37:02.419962 18320 raft_consensus.cc:3037] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:02.419962 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9bea116d64c4ab03e20ad33cbf446" candidate_uuid: "14f9ab5a180d45b5862663e07fdc033a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad0955309284c17a3fa197974176e69"
I20260927 18:37:02.420226 18189 raft_consensus.cc:3037] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:02.422376 18320 raft_consensus.cc:2454] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f9ab5a180d45b5862663e07fdc033a in term 1.
I20260927 18:37:02.422740 18383 leader_election.cc:302] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f9ab5a180d45b5862663e07fdc033a, 40a7159299af4f4b9cb9d4a89e96813a; no voters:
I20260927 18:37:02.422909 18189 raft_consensus.cc:2454] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f9ab5a180d45b5862663e07fdc033a in term 1.
I20260927 18:37:02.423005 18520 raft_consensus.cc:2781] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:02.423506 18520 raft_consensus.cc:686] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 1 LEADER]: Becoming Leader. State: Replica: 14f9ab5a180d45b5862663e07fdc033a, State: Running, Role: LEADER
I20260927 18:37:02.423993 18520 consensus_queue.cc:237] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } }
I20260927 18:37:02.428360 18047 catalog_manager.cc:5495] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f9ab5a180d45b5862663e07fdc033a (127.0.97.67). New cstate: current_term: 1 leader_uuid: "14f9ab5a180d45b5862663e07fdc033a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f9ab5a180d45b5862663e07fdc033a" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33499 } health_report { overall_health: HEALTHY } } }
I20260927 18:37:02.450201 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.444597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.0.97.67:33499)
0927 18:37:02.444821 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.444828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.444893 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:37:02.445143 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.445147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.445230 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.445380 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:02.445741 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.446555 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.448023 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.448463 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.448603 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.449108 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:02.449596 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:02.449770 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.449845 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.449930 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 18:37:02.458402 18320 raft_consensus.cc:1261] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 1 FOLLOWER]: Refusing update from remote peer 14f9ab5a180d45b5862663e07fdc033a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:02.458400 18189 raft_consensus.cc:1261] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 1 FOLLOWER]: Refusing update from remote peer 14f9ab5a180d45b5862663e07fdc033a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:02.459443 18523 consensus_queue.cc:1035] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad0955309284c17a3fa197974176e69" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:02.459852 18520 consensus_queue.cc:1035] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a7159299af4f4b9cb9d4a89e96813a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:02.473596 18528 mvcc.cc:205] Tried to move back new op lower bound from 7334028175178907648 to 7334028175052324864. Current Snapshot: MvccSnapshot[applied={T|T < 7334028175178907648}]
I20260927 18:37:02.473994 18526 mvcc.cc:205] Tried to move back new op lower bound from 7334028175178907648 to 7334028175052324864. Current Snapshot: MvccSnapshot[applied={T|T < 7334028175178907648 or (T in {7334028175178907648})}]
I20260927 18:37:02.487881 18048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:02.488188 18048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:02.491891 18048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c5971a6cbaa49258617b4a1f43055f6: Sending DeleteTablet for 3 replicas of tablet a8a9bea116d64c4ab03e20ad33cbf446
I20260927 18:37:02.493019 18300 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:02 UTC) from {username='slave'} at 127.0.0.1:47014
I20260927 18:37:02.493124 18169 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:02 UTC) from {username='slave'} at 127.0.0.1:44572
I20260927 18:37:02.493741 18430 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9bea116d64c4ab03e20ad33cbf446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:02 UTC) from {username='slave'} at 127.0.0.1:38686
I20260927 18:37:02.495748 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18107
I20260927 18:37:02.503134 18538 tablet_replica.cc:330] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: stopping tablet replica
I20260927 18:37:02.503425 18536 tablet_replica.cc:330] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: stopping tablet replica
I20260927 18:37:02.503577 18538 raft_consensus.cc:2229] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:02.503948 18536 raft_consensus.cc:2229] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:02.504582 18536 raft_consensus.cc:2258] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:02.505709 18538 raft_consensus.cc:2258] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:02.505975 18537 tablet_replica.cc:330] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: stopping tablet replica
I20260927 18:37:02.506894 18537 raft_consensus.cc:2229] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:02.507404 18537 raft_consensus.cc:2258] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:02.507997 18538 ts_tablet_manager.cc:1905] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:02.509060 18536 ts_tablet_manager.cc:1905] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:02.513559 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.497261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.0.97.65:39721)
0927 18:37:02.497483 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.497493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.505774 (+ 8281us) server_negotiation.cc:407] Connection header received
0927 18:37:02.505872 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.505877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.505983 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.506096 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:02.506547 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.507815 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.509693 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.510310 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.510409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.511880 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:02.511890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:02.511905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:02.512675 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:02.512735 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:02.512758 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:02.512843 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.513093 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.513211 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:02.514890 18229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:02.517089 18538 ts_tablet_manager.cc:1918] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:02.517310 18538 log.cc:1196] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/a8a9bea116d64c4ab03e20ad33cbf446
I20260927 18:37:02.518049 18538 ts_tablet_manager.cc:1939] T a8a9bea116d64c4ab03e20ad33cbf446 P 14f9ab5a180d45b5862663e07fdc033a: Deleting consensus metadata
I20260927 18:37:02.518451 18536 ts_tablet_manager.cc:1918] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:02.518644 18536 log.cc:1196] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/a8a9bea116d64c4ab03e20ad33cbf446
I20260927 18:37:02.519358 18536 ts_tablet_manager.cc:1939] T a8a9bea116d64c4ab03e20ad33cbf446 P 40a7159299af4f4b9cb9d4a89e96813a: Deleting consensus metadata
I20260927 18:37:02.520246 18034 catalog_manager.cc:4841] TS 14f9ab5a180d45b5862663e07fdc033a (127.0.97.67:33499): tablet a8a9bea116d64c4ab03e20ad33cbf446 (table test-table [id=c96758ee5ab54dc494b3438ca0c8b681]) successfully deleted
I20260927 18:37:02.521193 18032 catalog_manager.cc:4841] TS 40a7159299af4f4b9cb9d4a89e96813a (127.0.97.66:39249): tablet a8a9bea116d64c4ab03e20ad33cbf446 (table test-table [id=c96758ee5ab54dc494b3438ca0c8b681]) successfully deleted
I20260927 18:37:02.515406 18537 ts_tablet_manager.cc:1905] T a8a9bea116d64c4ab03e20ad33cbf446 P 0ad0955309284c17a3fa197974176e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:02.621465 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18107
I20260927 18:37:02.639611 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18238
W20260927 18:37:02.639638 18035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:39721 (error 108)
I20260927 18:37:02.641758 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.641293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44594)
0927 18:37:02.641459 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.641580 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:37:02.642042 18035 catalog_manager.cc:4584] TS 0ad0955309284c17a3fa197974176e69 (127.0.97.65:39721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8a9bea116d64c4ab03e20ad33cbf446: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
I20260927 18:37:02.647231 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.640518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.97.66:39249)
0927 18:37:02.640701 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.640708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.640803 (+ 95us) server_negotiation.cc:407] Connection header received
0927 18:37:02.640960 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.640966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.641081 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.641203 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:02.641716 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.642662 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.644062 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.644472 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.644612 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.645747 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:02.645755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:02.645769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:02.646419 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:02.646463 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:02.646487 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:02.646608 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.646717 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.646821 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:37:02.647984 18360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:02.696031 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.695500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44604)
0927 18:37:02.695739 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.695852 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:37:02.697216 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.696818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44608)
0927 18:37:02.696981 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.697078 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:37:02.740787 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18238
I20260927 18:37:02.758023 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.757427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44610)
0927 18:37:02.757635 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.757718 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:37:02.759068 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.758734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44624)
0927 18:37:02.758881 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.758957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:37:02.763820 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18368
I20260927 18:37:02.770887 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.764722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.97.67:33499)
0927 18:37:02.764980 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:02.764986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:02.765004 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:37:02.765064 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:02.765071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:02.765152 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:02.765282 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:02.765657 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.766522 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.767794 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:02.768234 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:02.768379 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:02.769467 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:02.769478 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:02.769493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:02.770127 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:02.770178 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:02.770200 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:02.770321 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:02.770463 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:37:02.770558 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:37:02.771643 18490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:02.840209 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.839612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44638)
0927 18:37:02.839932 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.840034 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260927 18:37:02.841356 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:02.840984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44640)
0927 18:37:02.841174 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:02.841257 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 18:37:02.893308 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18368
I20260927 18:37:02.909365 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18017
I20260927 18:37:02.910526 18078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:03.020332 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18017
I20260927 18:37:03.020433 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:03.019843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44652)
0927 18:37:03.020030 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:03.020143 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:37:03.021613 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:03.021141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39721 (local address 127.0.0.1:44654)
0927 18:37:03.021348 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:03.021445 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
2026-09-27T18:37:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:03 dist-test-slave-n3p4 krb5kdc[18547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:03 dist-test-slave-n3p4 krb5kdc[18547](info): set up 2 sockets
Sep 27 18:37:03 dist-test-slave-n3p4 krb5kdc[18547](info): commencing operation
krb5kdc: starting...
W20260927 18:37:04.145884 389 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:04 dist-test-slave-n3p4 krb5kdc[18547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:04.384508 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43413
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:04.717180 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:04.717501 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:04.717682 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:04.724882 18560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:04.725031 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:04.725138 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:04.725244 18560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:04.725312 18560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:04.735352 18560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:04.736675 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:04.738086 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:04.750581 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:04.750669 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:04.750852 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:04.751528 18560 server_base.cc:1034] running on GCE node
I20260927 18:37:04.752207 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:04.753494 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:04.754685 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790534224754645 us; error 33 us; skew 500 ppm
I20260927 18:37:04.755038 18560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:04 dist-test-slave-n3p4 krb5kdc[18547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:04.758154 18560 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:04.759549 18560 webserver.cc:460] Webserver started at http://127.0.97.126:40671/ using document root <none> and password file <none>
I20260927 18:37:04.760211 18560 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:04.760361 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:04.760661 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:04.763592 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "383b06ffd889459ead4fd08e2f277cac"
format_stamp: "Formatted at 2026-09-27 18:37:04 on dist-test-slave-n3p4"
server_key: "541da2f785fc8dc8872a8cd80fe127eb"
server_key_iv: "c2fe9cee5b4897a18108fd9d4753ce27"
server_key_version: "encryptionkey@0"
I20260927 18:37:04.764367 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "383b06ffd889459ead4fd08e2f277cac"
format_stamp: "Formatted at 2026-09-27 18:37:04 on dist-test-slave-n3p4"
server_key: "541da2f785fc8dc8872a8cd80fe127eb"
server_key_iv: "c2fe9cee5b4897a18108fd9d4753ce27"
server_key_version: "encryptionkey@0"
I20260927 18:37:04.769647 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:37:04.773202 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:04.774740 18560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:04.774928 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "383b06ffd889459ead4fd08e2f277cac"
format_stamp: "Formatted at 2026-09-27 18:37:04 on dist-test-slave-n3p4"
server_key: "541da2f785fc8dc8872a8cd80fe127eb"
server_key_iv: "c2fe9cee5b4897a18108fd9d4753ce27"
server_key_version: "encryptionkey@0"
I20260927 18:37:04.775132 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:04.787966 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:04.791805 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:04.792071 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:04.810592 18560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43413
I20260927 18:37:04.810616 18626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43413 every 8 connection(s)
I20260927 18:37:04.812007 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:04.815864 18627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:04.817451 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18560
I20260927 18:37:04.817674 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:04.818167 389 external_mini_cluster.cc:1428] Setting key 7e3788ddafd6a7e2ad00a6f225cb0dc1
I20260927 18:37:04.830622 18627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: Bootstrap starting.
Sep 27 18:37:04 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:04.833652 18627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:04.835042 18627 log.cc:824] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:04.838461 18627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: No bootstrap required, opened a new log
I20260927 18:37:04.842571 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:04.820357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.0.97.126:43413)
0927 18:37:04.821189 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:04.821204 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:04.821268 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:37:04.823045 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:04.823083 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:04.823616 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:04.824051 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:04.825436 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:04.827501 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:04.828870 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:04.829892 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:04.830067 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:04.834036 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:04.834052 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:04.834073 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:04.834131 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:04.838140 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:04.838873 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:04.838883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:04.838890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:04.839122 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:04.839456 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:04.839461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:04.839464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:04.840116 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:04.840451 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:04.840736 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:37:04.841175 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":332,"threads_started":1}
I20260927 18:37:04.847498 18627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } }
I20260927 18:37:04.847926 18627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:04.848062 18627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383b06ffd889459ead4fd08e2f277cac, State: Initialized, Role: FOLLOWER
I20260927 18:37:04.848706 18627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } }
I20260927 18:37:04.848984 18627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:04.849131 18627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:04.849313 18627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:04.851444 18627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } }
I20260927 18:37:04.851917 18627 leader_election.cc:302] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 383b06ffd889459ead4fd08e2f277cac; no voters:
I20260927 18:37:04.852474 18627 leader_election.cc:288] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:04.852595 18632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:04.853291 18632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [term 1 LEADER]: Becoming Leader. State: Replica: 383b06ffd889459ead4fd08e2f277cac, State: Running, Role: LEADER
I20260927 18:37:04.853881 18632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } }
I20260927 18:37:04.854590 18627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:04.856585 18634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383b06ffd889459ead4fd08e2f277cac. Latest consensus state: current_term: 1 leader_uuid: "383b06ffd889459ead4fd08e2f277cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } } }
I20260927 18:37:04.856792 18633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "383b06ffd889459ead4fd08e2f277cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b06ffd889459ead4fd08e2f277cac" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43413 } } }
I20260927 18:37:04.857055 18634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:04.857092 18633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:04.858950 18639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:04.864723 18639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:04.872857 18639 catalog_manager.cc:1347] Generated new cluster ID: 2061596d1c1348618958c0df6fc4c30f
I20260927 18:37:04.872973 18639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:04.890919 18639 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:04.892249 18639 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:04.900162 18639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: Generated new TSK 0
I20260927 18:37:04.900938 18639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:37:04.966881 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:05.297442 18653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:05.297724 18653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:05.297914 18653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:05.305523 18653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:05.305624 18653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:05.305889 18653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:37:05.316337 18653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:05.317624 18653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:05.318959 18653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:05.331902 18661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:05.331902 18658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:05.331902 18659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:05.333127 18653 server_base.cc:1034] running on GCE node
I20260927 18:37:05.333920 18653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:05.334908 18653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:05.336202 18653 hybrid_clock.cc:648] HybridClock initialized: now 1790534225336137 us; error 51 us; skew 500 ppm
I20260927 18:37:05.336611 18653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:05 dist-test-slave-n3p4 krb5kdc[18547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:05.340004 18653 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:37:05.341536 18653 webserver.cc:460] Webserver started at http://127.0.97.65:38993/ using document root <none> and password file <none>
I20260927 18:37:05.342268 18653 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:05.342367 18653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:05.342610 18653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:05.345428 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "147a6efeedd44fb88956b894841c707d"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "c95ba353f31d497c17ebb7fd09165d61"
server_key_iv: "b77524f5c3702448948722b768bd6026"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.346271 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "147a6efeedd44fb88956b894841c707d"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "c95ba353f31d497c17ebb7fd09165d61"
server_key_iv: "b77524f5c3702448948722b768bd6026"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.351310 18653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:37:05.355429 18668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.356943 18653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:05.357121 18653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "147a6efeedd44fb88956b894841c707d"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "c95ba353f31d497c17ebb7fd09165d61"
server_key_iv: "b77524f5c3702448948722b768bd6026"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.357321 18653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:05.420032 18653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:05.424305 18653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:05.424606 18653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:05.425581 18653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:05.426963 18653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:05.427068 18653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.427225 18653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:05.427301 18653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.462553 18653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41017
I20260927 18:37:05.462647 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41017 every 8 connection(s)
I20260927 18:37:05.464217 18653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:05.471792 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18653
I20260927 18:37:05.472000 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:05.472607 389 external_mini_cluster.cc:1428] Setting key e3718979d93763563dc19dd7233c774b
Sep 27 18:37:05 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:05.487025 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:05.467042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60783 (local address 127.0.97.126:43413)
0927 18:37:05.467534 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:05.467538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:05.468634 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 18:37:05.470621 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:05.470626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:05.470715 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:05.470848 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:05.472913 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.473648 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.475532 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.476302 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.476458 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:05.480267 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:05.480276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:05.480283 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:05.480330 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:05.483714 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:05.484650 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:05.484654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:05.484656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:05.484820 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:05.485301 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:05.485304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:05.485306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:05.485555 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:05.485661 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:05.486571 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 18:37:05.486770 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260927 18:37:05.488535 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:05.467660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43413 (local address 127.0.97.65:60783)
0927 18:37:05.468444 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:05.468490 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:05.470337 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:05.471092 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:05.471100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:05.471794 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:05.472719 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:05.472750 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.473824 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.473829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:05.475382 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:05.475395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.476541 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.476547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:05.476753 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:05.477445 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:05.477466 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:05.480038 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:05.483852 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:05.483862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:05.483883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:05.484527 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:05.484935 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:05.484943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:05.484949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:05.485190 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:05.485673 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:05.485683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:05.486151 (+ 468us) client_negotiation.cc:769] Sending connection context
0927 18:37:05.486467 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 18:37:05.487082 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":292,"threads_started":1}
I20260927 18:37:05.490108 18781 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43413
I20260927 18:37:05.490420 18781 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:05.491127 18781 heartbeater.cc:511] Master 127.0.97.126:43413 requested a full tablet report, sending...
I20260927 18:37:05.493132 18592 ts_manager.cc:194] Registered new tserver with Master: 147a6efeedd44fb88956b894841c707d (127.0.97.65:41017)
I20260927 18:37:05.494621 18592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:60783
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:37:05.529516 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:05.854985 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:05.855263 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:05.855451 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:05.863052 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:05.863164 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:05.863324 18787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:37:05.872800 18787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:05.874034 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:05.875274 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:05.887676 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:05.887769 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:05.887722 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:05.888633 18787 server_base.cc:1034] running on GCE node
I20260927 18:37:05.889263 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:05.890255 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:05.891479 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790534225891435 us; error 38 us; skew 500 ppm
I20260927 18:37:05.891814 18787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:05 dist-test-slave-n3p4 krb5kdc[18547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:05.900254 18787 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:37:05.901914 18787 webserver.cc:460] Webserver started at http://127.0.97.66:43753/ using document root <none> and password file <none>
I20260927 18:37:05.902637 18787 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:05.902787 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:05.903128 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:05.906138 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "810b98ac69761759c04ca6de76b7fa5f"
server_key_iv: "9651b917d82ed3a547f761a5b971c474"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.906948 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "810b98ac69761759c04ca6de76b7fa5f"
server_key_iv: "9651b917d82ed3a547f761a5b971c474"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.912465 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 18:37:05.916003 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.917340 18787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:05.917506 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431"
format_stamp: "Formatted at 2026-09-27 18:37:05 on dist-test-slave-n3p4"
server_key: "810b98ac69761759c04ca6de76b7fa5f"
server_key_iv: "9651b917d82ed3a547f761a5b971c474"
server_key_version: "encryptionkey@0"
I20260927 18:37:05.917713 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:05.939054 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:05.943200 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:05.943475 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:05.944406 18787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:05.945657 18787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:05.945767 18787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.945950 18787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:05.946022 18787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:05.979576 18787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:45987
I20260927 18:37:05.979620 18914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:45987 every 8 connection(s)
I20260927 18:37:05.980872 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:05.982437 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18787
I20260927 18:37:05.982610 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:05.983145 389 external_mini_cluster.cc:1428] Setting key ab21b286435c3d73ea668cf45c9dd075
Sep 27 18:37:05 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:06.005525 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:05.983777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56963 (local address 127.0.97.126:43413)
0927 18:37:05.984015 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:05.984022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:05.985634 (+ 1612us) server_negotiation.cc:407] Connection header received
0927 18:37:05.988610 (+ 2976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:05.988614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:05.988680 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:05.988789 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:05.990870 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.991564 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.994054 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.994541 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.994721 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:05.998081 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:05.998091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:05.998098 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:05.998146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:06.001911 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.002775 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.002784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.002790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.002966 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.003398 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.003406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.003412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.003873 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:06.004041 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.004734 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.005015 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:37:06.006072 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:05.984536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43413 (local address 127.0.97.66:56963)
0927 18:37:05.985450 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:05.985512 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:05.988309 (+ 2797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:05.989046 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:05.989060 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:05.989887 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:05.990678 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:05.990705 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.991721 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.991729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:05.993886 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:05.993903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:05.994774 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:05.994781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:05.994971 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:05.995650 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:05.995677 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:05.997819 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:06.002044 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:06.002051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:06.002062 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:06.002629 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:06.003077 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:06.003080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:06.003082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:06.003270 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:06.004002 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:06.004008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:06.004323 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.004541 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.004921 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260927 18:37:06.007637 18915 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43413
I20260927 18:37:06.007942 18915 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:06.008632 18915 heartbeater.cc:511] Master 127.0.97.126:43413 requested a full tablet report, sending...
I20260927 18:37:06.010142 18592 ts_manager.cc:194] Registered new tserver with Master: f5dc2bb9673e4eef8c2ef4b08efcc431 (127.0.97.66:45987)
I20260927 18:37:06.010965 18592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:56963
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:37:06.041199 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:06.360949 18920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:06.361202 18920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:06.361394 18920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:06.368635 18920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:06.368743 18920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:06.368937 18920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:37:06.378760 18920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:06.380017 18920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:06.381285 18920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:06.394670 18926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:06.394743 18925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:06.394767 18928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:06.395328 18920 server_base.cc:1034] running on GCE node
I20260927 18:37:06.395939 18920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:06.396835 18920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:06.398061 18920 hybrid_clock.cc:648] HybridClock initialized: now 1790534226398016 us; error 54 us; skew 500 ppm
I20260927 18:37:06.398376 18920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:06 dist-test-slave-n3p4 krb5kdc[18547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:06.401423 18920 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:37:06.403060 18920 webserver.cc:460] Webserver started at http://127.0.97.67:33375/ using document root <none> and password file <none>
I20260927 18:37:06.403798 18920 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:06.403927 18920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:06.404278 18920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:06.407150 18920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "af7de45579be4f42b3f0eeb002a1654f"
format_stamp: "Formatted at 2026-09-27 18:37:06 on dist-test-slave-n3p4"
server_key: "cd343cfb52c60f93bd12763ef848ba73"
server_key_iv: "88bb96b2d32402ed710d979ad8e65b80"
server_key_version: "encryptionkey@0"
I20260927 18:37:06.407912 18920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "af7de45579be4f42b3f0eeb002a1654f"
format_stamp: "Formatted at 2026-09-27 18:37:06 on dist-test-slave-n3p4"
server_key: "cd343cfb52c60f93bd12763ef848ba73"
server_key_iv: "88bb96b2d32402ed710d979ad8e65b80"
server_key_version: "encryptionkey@0"
I20260927 18:37:06.412773 18920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 18:37:06.416100 18935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:06.417484 18920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:06.417760 18920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "af7de45579be4f42b3f0eeb002a1654f"
format_stamp: "Formatted at 2026-09-27 18:37:06 on dist-test-slave-n3p4"
server_key: "cd343cfb52c60f93bd12763ef848ba73"
server_key_iv: "88bb96b2d32402ed710d979ad8e65b80"
server_key_version: "encryptionkey@0"
I20260927 18:37:06.418075 18920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:06.432094 18920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:06.436410 18920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:06.436750 18920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:06.437762 18920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:06.439278 18920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:06.439360 18920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:06.439524 18920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:06.439603 18920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:06.478694 18920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43741
I20260927 18:37:06.478718 19047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43741 every 8 connection(s)
I20260927 18:37:06.480120 18920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:06.483528 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 18920
I20260927 18:37:06.483728 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:06.484221 389 external_mini_cluster.cc:1428] Setting key e71e16d178ec25b997385c14d2629059
Sep 27 18:37:06 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:06.497522 18781 heartbeater.cc:503] Master 127.0.97.126:43413 was elected leader, sending a full tablet report...
I20260927 18:37:06.500839 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.482964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56723 (local address 127.0.97.126:43413)
0927 18:37:06.483203 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.483210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.484481 (+ 1271us) server_negotiation.cc:407] Connection header received
0927 18:37:06.486673 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.486679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.486798 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.486909 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:06.488417 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.489171 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.490760 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.491172 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.491270 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.494520 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:06.494525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:06.494528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:06.494558 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:06.497115 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.498187 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.498193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.498196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.498360 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.498922 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.498934 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.498936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.499293 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:06.499436 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.500244 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.500481 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:37:06.501766 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.483437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43413 (local address 127.0.97.67:56723)
0927 18:37:06.484297 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.484351 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.486419 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.487212 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.487220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.487738 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:06.488271 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.488289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.489312 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.489316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.490616 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.490628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.491356 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.491360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.491544 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.492250 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:06.492271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:06.494351 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:06.497293 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:06.497302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:06.497316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:06.497976 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:06.498512 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:06.498517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:06.498521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:06.498778 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:06.499501 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:06.499508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:06.499888 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.500137 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.500663 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":398,"threads_started":1}
I20260927 18:37:06.503217 19048 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43413
I20260927 18:37:06.503566 19048 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:06.504284 19048 heartbeater.cc:511] Master 127.0.97.126:43413 requested a full tablet report, sending...
I20260927 18:37:06.505600 18592 ts_manager.cc:194] Registered new tserver with Master: af7de45579be4f42b3f0eeb002a1654f (127.0.97.67:43741)
I20260927 18:37:06.506445 18592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:56723
I20260927 18:37:06.511991 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:06.529634 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.516199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.0.97.126:43413)
0927 18:37:06.516396 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.516400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.516884 (+ 484us) server_negotiation.cc:407] Connection header received
0927 18:37:06.516996 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.517000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.517060 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.517144 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:06.518655 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.519381 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.520802 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.521418 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.521516 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.524124 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:06.524129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:06.524132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:06.524161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:06.526860 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.527574 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.527578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.527580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.527698 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.528169 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.528173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.528175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.528488 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:06.528776 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.529133 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.529338 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 18:37:06.533443 18592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:06.540271 18592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:06.576618 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.552301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:45987 (local address 127.0.0.1:38440)
0927 18:37:06.567328 (+ 15027us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.567354 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.567507 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.569052 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.569059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.569086 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:06.569528 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.569542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.571519 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.571526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.574278 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.574290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.575606 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.575612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.575821 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.575883 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.576021 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.576231 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14839,"thread_start_us":15702,"threads_started":1}
I20260927 18:37:06.576604 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.552358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.97.66:45987)
0927 18:37:06.568036 (+ 15678us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.568046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.568080 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:37:06.568185 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.568197 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.568519 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.568735 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:06.569738 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.571373 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.574436 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.575401 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.575542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.575777 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.576075 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.576241 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15517,"thread_start_us":15423,"threads_started":1}
I20260927 18:37:06.579450 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.553047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.0.97.67:43741)
0927 18:37:06.570582 (+ 17535us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.570590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.570622 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:37:06.570713 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.570720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.570964 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.571149 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:06.572112 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.573690 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.576290 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.578298 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.578444 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.578677 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.578919 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.579125 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17361,"thread_start_us":17199,"threads_started":1}
I20260927 18:37:06.579435 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.552269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43741 (local address 127.0.0.1:51548)
0927 18:37:06.563746 (+ 11477us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.563792 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.563966 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.571337 (+ 7371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.571341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.571365 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:06.571842 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.571857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.573871 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.573879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.576104 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.576119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.578501 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.578507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.578704 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.578746 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.578875 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.579036 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11225,"mutex_wait_us":3,"thread_start_us":11078,"threads_started":1}
I20260927 18:37:06.580134 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.553553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.97.65:41017)
0927 18:37:06.570075 (+ 16522us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.570084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.571046 (+ 962us) server_negotiation.cc:407] Connection header received
0927 18:37:06.571345 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.571352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.571593 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.571812 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:06.572531 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.574095 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.576630 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.578297 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.578422 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.579178 (+ 756us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.579259 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.579407 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16341,"thread_start_us":16122,"threads_started":1}
I20260927 18:37:06.580206 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.552633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41017 (local address 127.0.0.1:51778)
0927 18:37:06.570926 (+ 18293us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.570941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.571171 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.571823 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.571826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.571852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:06.572326 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.572338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.574264 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.574269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.576437 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.576451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.578636 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.578643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.578808 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.578837 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.578922 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.579883 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18169,"mutex_wait_us":743,"thread_start_us":18035,"threads_started":1}
I20260927 18:37:06.581125 18850 tablet_service.cc:1467] Processing CreateTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 (DEFAULT_TABLE table=test-table [id=8152d783fa7940118469d6fad856550a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:06.582773 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dc4197ecb9425bb913d975a24d8db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:06.584080 18983 tablet_service.cc:1467] Processing CreateTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 (DEFAULT_TABLE table=test-table [id=8152d783fa7940118469d6fad856550a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:06.584079 18716 tablet_service.cc:1467] Processing CreateTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 (DEFAULT_TABLE table=test-table [id=8152d783fa7940118469d6fad856550a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:06.585510 18716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dc4197ecb9425bb913d975a24d8db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:06.585510 18983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dc4197ecb9425bb913d975a24d8db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:06.595703 19066 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Bootstrap starting.
I20260927 18:37:06.597580 19067 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Bootstrap starting.
I20260927 18:37:06.599292 19066 tablet_bootstrap.cc:654] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:06.600421 19066 log.cc:824] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:06.600958 19067 tablet_bootstrap.cc:654] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:06.602293 19067 log.cc:824] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:06.602566 19069 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Bootstrap starting.
I20260927 18:37:06.603696 19066 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: No bootstrap required, opened a new log
I20260927 18:37:06.603988 19066 ts_tablet_manager.cc:1397] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 18:37:06.605581 19067 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: No bootstrap required, opened a new log
I20260927 18:37:06.605871 19067 ts_tablet_manager.cc:1397] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 18:37:06.605944 19069 tablet_bootstrap.cc:654] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:06.607123 19069 log.cc:824] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:06.609948 19069 tablet_bootstrap.cc:492] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: No bootstrap required, opened a new log
I20260927 18:37:06.610188 19069 ts_tablet_manager.cc:1397] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 18:37:06.611784 19066 raft_consensus.cc:348] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.612133 19066 raft_consensus.cc:374] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:06.612231 19066 raft_consensus.cc:729] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5dc2bb9673e4eef8c2ef4b08efcc431, State: Initialized, Role: FOLLOWER
I20260927 18:37:06.612814 19066 consensus_queue.cc:260] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.614574 18915 heartbeater.cc:503] Master 127.0.97.126:43413 was elected leader, sending a full tablet report...
I20260927 18:37:06.616883 19066 ts_tablet_manager.cc:1428] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 18:37:06.616880 19067 raft_consensus.cc:348] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.617342 19067 raft_consensus.cc:374] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:06.617451 19067 raft_consensus.cc:729] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7de45579be4f42b3f0eeb002a1654f, State: Initialized, Role: FOLLOWER
I20260927 18:37:06.618168 19067 consensus_queue.cc:260] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.619272 19048 heartbeater.cc:503] Master 127.0.97.126:43413 was elected leader, sending a full tablet report...
I20260927 18:37:06.620026 19067 ts_tablet_manager.cc:1428] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Time spent starting tablet: real 0.014s user 0.003s sys 0.009s
I20260927 18:37:06.621136 19069 raft_consensus.cc:348] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.621555 19069 raft_consensus.cc:374] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:06.621671 19069 raft_consensus.cc:729] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147a6efeedd44fb88956b894841c707d, State: Initialized, Role: FOLLOWER
I20260927 18:37:06.622290 19069 consensus_queue.cc:260] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.624074 19069 ts_tablet_manager.cc:1428] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 18:37:06.716676 18782 tablet.cc:2369] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:06.732007 19049 tablet.cc:2369] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:06.732858 18916 tablet.cc:2369] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:06.779726 19072 raft_consensus.cc:482] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:06.780061 19072 raft_consensus.cc:504] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.781766 19072 leader_election.cc:288] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af7de45579be4f42b3f0eeb002a1654f (127.0.97.67:43741), 147a6efeedd44fb88956b894841c707d (127.0.97.65:41017)
I20260927 18:37:06.788838 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.782215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56741 (local address 127.0.97.65:41017)
0927 18:37:06.782450 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.782455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.783278 (+ 823us) server_negotiation.cc:407] Connection header received
0927 18:37:06.783481 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.783486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.783544 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.783633 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:06.784247 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.784947 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.787111 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.787870 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.788027 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.788144 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.788437 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.788564 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:37:06.788888 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.782125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56135 (local address 127.0.97.67:43741)
0927 18:37:06.782360 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.782368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.783196 (+ 828us) server_negotiation.cc:407] Connection header received
0927 18:37:06.783410 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.783415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.783476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.783574 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:06.784242 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.784975 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.787130 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.787920 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.788069 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.788189 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.788479 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.788616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:37:06.788982 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.782191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41017 (local address 127.0.97.66:56741)
0927 18:37:06.783174 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.783188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.783342 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.783659 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.783667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.783691 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:06.784098 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.784107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.785106 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.785113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.786958 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.786970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.788113 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.788120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.788284 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.788308 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.788412 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.788543 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":107,"thread_start_us":605,"threads_started":1}
I20260927 18:37:06.789410 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.782045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43741 (local address 127.0.97.66:56135)
0927 18:37:06.783077 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:06.783093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:06.783259 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:06.783587 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:06.783591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:06.783613 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:06.784085 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.784095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.785130 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.785139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.786962 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:06.786974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.788152 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.788158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:06.788325 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.788348 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:37:06.788450 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:37:06.788571 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":6,"spinlock_wait_cycles":21760,"thread_start_us":701,"threads_started":1}
I20260927 18:37:06.790046 18736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dc4197ecb9425bb913d975a24d8db3" candidate_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147a6efeedd44fb88956b894841c707d" is_pre_election: true
I20260927 18:37:06.790397 19003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dc4197ecb9425bb913d975a24d8db3" candidate_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7de45579be4f42b3f0eeb002a1654f" is_pre_election: true
I20260927 18:37:06.790548 18736 raft_consensus.cc:2454] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5dc2bb9673e4eef8c2ef4b08efcc431 in term 0.
I20260927 18:37:06.790879 19003 raft_consensus.cc:2454] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5dc2bb9673e4eef8c2ef4b08efcc431 in term 0.
I20260927 18:37:06.791224 18803 leader_election.cc:302] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 147a6efeedd44fb88956b894841c707d, f5dc2bb9673e4eef8c2ef4b08efcc431; no voters:
I20260927 18:37:06.791604 19072 raft_consensus.cc:2781] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:06.791761 19072 raft_consensus.cc:482] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:06.791857 19072 raft_consensus.cc:3037] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:06.794067 19072 raft_consensus.cc:504] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.794667 19072 leader_election.cc:288] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [CANDIDATE]: Term 1 election: Requested vote from peers af7de45579be4f42b3f0eeb002a1654f (127.0.97.67:43741), 147a6efeedd44fb88956b894841c707d (127.0.97.65:41017)
I20260927 18:37:06.795167 18736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dc4197ecb9425bb913d975a24d8db3" candidate_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147a6efeedd44fb88956b894841c707d"
I20260927 18:37:06.795149 19003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dc4197ecb9425bb913d975a24d8db3" candidate_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7de45579be4f42b3f0eeb002a1654f"
I20260927 18:37:06.795375 19003 raft_consensus.cc:3037] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:06.795375 18736 raft_consensus.cc:3037] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:06.798642 19003 raft_consensus.cc:2454] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5dc2bb9673e4eef8c2ef4b08efcc431 in term 1.
I20260927 18:37:06.798642 18736 raft_consensus.cc:2454] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5dc2bb9673e4eef8c2ef4b08efcc431 in term 1.
I20260927 18:37:06.799084 18803 leader_election.cc:302] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 147a6efeedd44fb88956b894841c707d, f5dc2bb9673e4eef8c2ef4b08efcc431; no voters:
I20260927 18:37:06.799417 19072 raft_consensus.cc:2781] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:06.799909 19072 raft_consensus.cc:686] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 1 LEADER]: Becoming Leader. State: Replica: f5dc2bb9673e4eef8c2ef4b08efcc431, State: Running, Role: LEADER
I20260927 18:37:06.800458 19072 consensus_queue.cc:237] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } }
I20260927 18:37:06.805071 18592 catalog_manager.cc:5495] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5dc2bb9673e4eef8c2ef4b08efcc431 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5dc2bb9673e4eef8c2ef4b08efcc431" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 } health_report { overall_health: UNKNOWN } } }
I20260927 18:37:06.870669 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.865553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.0.97.66:45987)
0927 18:37:06.865739 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.865744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.865762 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:37:06.865964 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.865968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.866043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.866164 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:06.866535 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.867277 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.868650 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.869033 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.869120 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.869597 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:06.870158 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:06.870286 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.870422 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.870460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:37:06.878496 18736 raft_consensus.cc:1261] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 1 FOLLOWER]: Refusing update from remote peer f5dc2bb9673e4eef8c2ef4b08efcc431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:06.878576 19003 raft_consensus.cc:1261] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 1 FOLLOWER]: Refusing update from remote peer f5dc2bb9673e4eef8c2ef4b08efcc431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:06.879527 19077 consensus_queue.cc:1035] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7de45579be4f42b3f0eeb002a1654f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:06.879968 19072 consensus_queue.cc:1035] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "147a6efeedd44fb88956b894841c707d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:06.889904 19080 mvcc.cc:205] Tried to move back new op lower bound from 7334028193284616192 to 7334028192979136512. Current Snapshot: MvccSnapshot[applied={T|T < 7334028193284616192}]
I20260927 18:37:06.890684 19081 mvcc.cc:205] Tried to move back new op lower bound from 7334028193284616192 to 7334028192979136512. Current Snapshot: MvccSnapshot[applied={T|T < 7334028193284616192}]
I20260927 18:37:06.895782 19082 mvcc.cc:205] Tried to move back new op lower bound from 7334028193284616192 to 7334028192979136512. Current Snapshot: MvccSnapshot[applied={T|T < 7334028193284616192}]
I20260927 18:37:06.904557 18591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:06.904887 18591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:06.909226 18591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 383b06ffd889459ead4fd08e2f277cac: Sending DeleteTablet for 3 replicas of tablet 42dc4197ecb9425bb913d975a24d8db3
I20260927 18:37:06.910557 18716 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:06 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51778
I20260927 18:37:06.910533 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:06 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38440
I20260927 18:37:06.910529 18983 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dc4197ecb9425bb913d975a24d8db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:06 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51548
I20260927 18:37:06.916494 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18653
I20260927 18:37:06.923266 19089 tablet_replica.cc:330] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: stopping tablet replica
I20260927 18:37:06.923920 19090 tablet_replica.cc:330] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: stopping tablet replica
I20260927 18:37:06.924387 19090 raft_consensus.cc:2229] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:06.925192 19090 raft_consensus.cc:2258] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:06.926306 19089 raft_consensus.cc:2229] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:06.927011 19089 raft_consensus.cc:2258] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:06.927944 19091 tablet_replica.cc:330] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: stopping tablet replica
I20260927 18:37:06.928507 19091 raft_consensus.cc:2229] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:06.929440 19089 ts_tablet_manager.cc:1905] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:06.929666 19091 raft_consensus.cc:2258] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:06.932219 19091 ts_tablet_manager.cc:1905] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:06.932255 19090 ts_tablet_manager.cc:1905] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:06.937219 19089 ts_tablet_manager.cc:1918] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:06.937397 19089 log.cc:1196] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/42dc4197ecb9425bb913d975a24d8db3
I20260927 18:37:06.938019 19089 ts_tablet_manager.cc:1939] T 42dc4197ecb9425bb913d975a24d8db3 P 147a6efeedd44fb88956b894841c707d: Deleting consensus metadata
I20260927 18:37:06.939599 19090 ts_tablet_manager.cc:1918] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:06.939772 19090 log.cc:1196] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/42dc4197ecb9425bb913d975a24d8db3
I20260927 18:37:06.940014 18576 catalog_manager.cc:4841] TS 147a6efeedd44fb88956b894841c707d (127.0.97.65:41017): tablet 42dc4197ecb9425bb913d975a24d8db3 (table test-table [id=8152d783fa7940118469d6fad856550a]) successfully deleted
I20260927 18:37:06.940371 19090 ts_tablet_manager.cc:1939] T 42dc4197ecb9425bb913d975a24d8db3 P f5dc2bb9673e4eef8c2ef4b08efcc431: Deleting consensus metadata
I20260927 18:37:06.942106 18578 catalog_manager.cc:4841] TS f5dc2bb9673e4eef8c2ef4b08efcc431 (127.0.97.66:45987): tablet 42dc4197ecb9425bb913d975a24d8db3 (table test-table [id=8152d783fa7940118469d6fad856550a]) successfully deleted
I20260927 18:37:06.943727 19091 ts_tablet_manager.cc:1918] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:06.943914 19091 log.cc:1196] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/42dc4197ecb9425bb913d975a24d8db3
I20260927 18:37:06.944340 19091 ts_tablet_manager.cc:1939] T 42dc4197ecb9425bb913d975a24d8db3 P af7de45579be4f42b3f0eeb002a1654f: Deleting consensus metadata
I20260927 18:37:06.945923 18579 catalog_manager.cc:4841] TS af7de45579be4f42b3f0eeb002a1654f (127.0.97.67:43741): tablet 42dc4197ecb9425bb913d975a24d8db3 (table test-table [id=8152d783fa7940118469d6fad856550a]) successfully deleted
Sep 27 18:37:06 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:37:06.955201 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:06.918207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.97.65:41017)
0927 18:37:06.918447 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:06.918456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:06.930599 (+ 12143us) server_negotiation.cc:407] Connection header received
0927 18:37:06.937509 (+ 6910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:06.937516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:06.937629 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:06.937710 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:06.940025 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.941173 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.943264 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:06.943913 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:06.944042 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:06.949379 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:06.949391 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:06.949405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:06.949437 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:06.952301 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.953129 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.953136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.953138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.953256 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:06.953652 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:06.953655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:06.953657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:06.954051 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:06.954273 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:06.954699 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 18:37:06.954884 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:37:06.955928 18776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:07.064370 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18653
I20260927 18:37:07.083066 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18787
Sep 27 18:37:07 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:37:07.100394 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:07.084030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.0.97.66:45987)
0927 18:37:07.084250 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:07.084262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:07.084284 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:37:07.084363 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:07.084369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:07.084448 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:07.084571 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:07.086225 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.087093 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.088933 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.089366 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.089456 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:07.094546 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:07.094559 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:07.094573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:07.094607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:07.097662 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:07.098459 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:07.098465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:07.098467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:07.098607 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:07.098993 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:07.098996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:07.098998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:07.099340 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:07.099576 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:07.099908 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 18:37:07.100057 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:37:07.101089 18910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:07.218421 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18787
I20260927 18:37:07.237916 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18920
Sep 27 18:37:07 dist-test-slave-n3p4 krb5kdc[18547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:37:07.253000 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:07.238713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.67:43741)
0927 18:37:07.238899 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:07.238906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:07.239006 (+ 100us) server_negotiation.cc:407] Connection header received
0927 18:37:07.239220 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:07.239232 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:07.239313 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:07.239433 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:07.240668 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.241407 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.242767 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.243156 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.243286 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:07.247605 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:07.247618 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:07.247635 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:07.247669 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:07.250608 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:07.251247 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:07.251254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:07.251256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:07.251379 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:07.251675 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:07.251678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:07.251680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:07.252009 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:07.252222 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:07.252448 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:37:07.252608 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:37:07.253645 19043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:07.342402 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18920
I20260927 18:37:07.359557 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 18560
I20260927 18:37:07.360921 18622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:07.476397 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 18560
2026-09-27T18:37:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4453 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:07Z Disabled control of system clock
I20260927 18:37:07.550295 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38283
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:07.877521 19100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:07.877815 19100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:07.877951 19100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:07.885195 19100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:07.885296 19100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:07.885366 19100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:07.885452 19100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:07.885515 19100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:07.894845 19100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:07.896158 19100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:07.898006 19100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:07.910243 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:07.910261 19106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:07.910342 19105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:07.911463 19100 server_base.cc:1034] running on GCE node
I20260927 18:37:07.912285 19100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:07.914014 19100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:07.915283 19100 hybrid_clock.cc:648] HybridClock initialized: now 1790534227915238 us; error 40 us; skew 500 ppm
I20260927 18:37:07.915603 19100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:07.918129 19100 webserver.cc:460] Webserver started at http://127.0.97.126:37175/ using document root <none> and password file <none>
I20260927 18:37:07.918802 19100 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:07.918889 19100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:07.919203 19100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:07.922362 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0"
format_stamp: "Formatted at 2026-09-27 18:37:07 on dist-test-slave-n3p4"
server_key: "18cc8be0eeb817555d551f94e4190136"
server_key_iv: "aa4721ed4679651c5a1830b8c43c32c5"
server_key_version: "encryptionkey@0"
I20260927 18:37:07.923149 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0"
format_stamp: "Formatted at 2026-09-27 18:37:07 on dist-test-slave-n3p4"
server_key: "18cc8be0eeb817555d551f94e4190136"
server_key_iv: "aa4721ed4679651c5a1830b8c43c32c5"
server_key_version: "encryptionkey@0"
I20260927 18:37:07.928450 19100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:37:07.931977 19114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:07.933401 19100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:07.933539 19100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0"
format_stamp: "Formatted at 2026-09-27 18:37:07 on dist-test-slave-n3p4"
server_key: "18cc8be0eeb817555d551f94e4190136"
server_key_iv: "aa4721ed4679651c5a1830b8c43c32c5"
server_key_version: "encryptionkey@0"
I20260927 18:37:07.933712 19100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:07.948176 19100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:07.948935 19100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:07.949173 19100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:07.967516 19100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38283
I20260927 18:37:07.967536 19165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38283 every 8 connection(s)
I20260927 18:37:07.968851 19100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:07.971026 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19100
I20260927 18:37:07.971210 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:07.971659 389 external_mini_cluster.cc:1428] Setting key 32e6a1cac4923d7f777f35bece332b1c
I20260927 18:37:07.973031 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:07.987043 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:07.973536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.0.97.126:38283)
0927 18:37:07.974681 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:07.974695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:07.974746 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:37:07.976509 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:07.976536 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:07.977141 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:07.977580 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:07.978138 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.980200 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.981576 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:07.982620 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:07.982817 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:07.983889 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:07.983901 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:07.983924 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:07.984694 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:07.984797 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:07.984822 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:07.984958 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:07.985165 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 18:37:07.985597 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":407,"threads_started":1}
I20260927 18:37:07.988152 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: Bootstrap starting.
I20260927 18:37:07.991466 19166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:07.992467 19166 log.cc:824] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:07.995180 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: No bootstrap required, opened a new log
I20260927 18:37:08.002467 19166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } }
I20260927 18:37:08.002800 19166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:08.002913 19166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac8fdbc6aa14f7bb91ca0de7d4840f0, State: Initialized, Role: FOLLOWER
I20260927 18:37:08.003448 19166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } }
I20260927 18:37:08.003680 19166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:08.003836 19166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:08.004011 19166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:08.006350 19166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } }
I20260927 18:37:08.006891 19166 leader_election.cc:302] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac8fdbc6aa14f7bb91ca0de7d4840f0; no voters:
I20260927 18:37:08.007561 19166 leader_election.cc:288] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:08.007704 19171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:08.008464 19171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac8fdbc6aa14f7bb91ca0de7d4840f0, State: Running, Role: LEADER
I20260927 18:37:08.008947 19171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } }
I20260927 18:37:08.009622 19166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:08.011765 19172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } } }
I20260927 18:37:08.012113 19172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:08.013546 19173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac8fdbc6aa14f7bb91ca0de7d4840f0. Latest consensus state: current_term: 1 leader_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac8fdbc6aa14f7bb91ca0de7d4840f0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38283 } } }
I20260927 18:37:08.014068 19173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:08.016192 19179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:08.020327 19179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:08.028698 19179 catalog_manager.cc:1347] Generated new cluster ID: 33216d92db2e4581a311f6c356c6e019
I20260927 18:37:08.028842 19179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:08.048549 19179 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:08.049927 19179 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:08.063704 19179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: Generated new TSK 0
I20260927 18:37:08.064769 19179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:08.083134 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--builtin_ntp_servers=127.0.97.84:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:08.414160 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:08.414505 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:08.414673 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:08.422187 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:08.422325 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:08.422502 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:37:08.431509 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:08.433274 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:08.435021 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:08.447971 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:08.448012 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:08.447932 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:08.448666 19190 server_base.cc:1034] running on GCE node
I20260927 18:37:08.449242 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:08.450259 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:08.451502 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790534228451468 us; error 46 us; skew 500 ppm
I20260927 18:37:08.451828 19190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:08.454064 19190 webserver.cc:460] Webserver started at http://127.0.97.65:43167/ using document root <none> and password file <none>
I20260927 18:37:08.454737 19190 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:08.454870 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:08.455197 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:08.458020 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "b0efa8306dda427897127b59df2abb23"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "1d63d53e241cfa9a35166cfeb10f896f"
server_key_iv: "42fce8c98408d040729b3a329e4e9982"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.458783 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "b0efa8306dda427897127b59df2abb23"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "1d63d53e241cfa9a35166cfeb10f896f"
server_key_iv: "42fce8c98408d040729b3a329e4e9982"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.463732 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:37:08.467036 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.468300 19190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:08.468472 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "b0efa8306dda427897127b59df2abb23"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "1d63d53e241cfa9a35166cfeb10f896f"
server_key_iv: "42fce8c98408d040729b3a329e4e9982"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.468686 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:08.491822 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:08.492684 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:08.492977 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:08.494017 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:08.495499 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:08.495623 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.495760 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:08.495849 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.529000 19190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:46861
I20260927 18:37:08.529086 19316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:46861 every 8 connection(s)
I20260927 18:37:08.530357 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:08.536928 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19190
I20260927 18:37:08.537091 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:08.537608 389 external_mini_cluster.cc:1428] Setting key 3749ff140e36d0b01f3c46d49b25a345
I20260927 18:37:08.540242 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--builtin_ntp_servers=127.0.97.84:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:08.543044 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:08.532936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41787 (local address 127.0.97.126:38283)
0927 18:37:08.533382 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:08.533386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:08.534596 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 18:37:08.536336 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:08.536340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:08.536403 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:08.536498 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:08.537699 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:08.538725 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:08.540224 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:08.540702 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:08.540845 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:08.542198 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:08.542204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:08.542208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:08.542233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:08.542272 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:08.542284 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:08.542388 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:08.542610 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 18:37:08.542699 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260927 18:37:08.543988 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:08.533539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38283 (local address 127.0.97.65:41787)
0927 18:37:08.534443 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:08.534483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:08.536100 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:08.536688 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:08.536695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:08.536814 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:08.537535 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:08.537552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:08.538881 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:08.538888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:08.540082 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:08.540093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:08.540930 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:08.540939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:08.541170 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:08.541894 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:08.541897 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:08.541932 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:08.542007 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:08.542386 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:08.542398 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:08.542408 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:37:08.542582 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:37:08.542952 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":312,"threads_started":1}
I20260927 18:37:08.545508 19317 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38283
I20260927 18:37:08.545867 19317 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:08.546586 19317 heartbeater.cc:511] Master 127.0.97.126:38283 requested a full tablet report, sending...
I20260927 18:37:08.548571 19131 ts_manager.cc:194] Registered new tserver with Master: b0efa8306dda427897127b59df2abb23 (127.0.97.65:46861)
I20260927 18:37:08.550107 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:41787
W20260927 18:37:08.880927 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:08.881261 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:08.881441 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:08.888967 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:08.889107 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:08.889303 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:37:08.898590 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:08.899907 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:08.901808 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:08.915761 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:08.915717 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:08.916013 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:08.916767 19321 server_base.cc:1034] running on GCE node
I20260927 18:37:08.917405 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:08.918416 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:08.919641 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790534228919583 us; error 56 us; skew 500 ppm
I20260927 18:37:08.919955 19321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:08.922237 19321 webserver.cc:460] Webserver started at http://127.0.97.66:37193/ using document root <none> and password file <none>
I20260927 18:37:08.922927 19321 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:08.923058 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:08.923389 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:08.926208 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "13b08c5d97e74cc4ae3fd7a4191611c7"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "8add94cb4d232f5abfc91e148f2d4312"
server_key_iv: "9cfd05b9ed80c019e316ab0d9ba97045"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.926990 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "13b08c5d97e74cc4ae3fd7a4191611c7"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "8add94cb4d232f5abfc91e148f2d4312"
server_key_iv: "9cfd05b9ed80c019e316ab0d9ba97045"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.931806 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:37:08.935101 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.936300 19321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:37:08.936473 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "13b08c5d97e74cc4ae3fd7a4191611c7"
format_stamp: "Formatted at 2026-09-27 18:37:08 on dist-test-slave-n3p4"
server_key: "8add94cb4d232f5abfc91e148f2d4312"
server_key_iv: "9cfd05b9ed80c019e316ab0d9ba97045"
server_key_version: "encryptionkey@0"
I20260927 18:37:08.936710 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:08.953378 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:08.954265 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:08.954545 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:08.955461 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:08.956905 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:08.957031 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.957165 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:08.957254 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:08.992398 19321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33767
I20260927 18:37:08.992451 19447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33767 every 8 connection(s)
I20260927 18:37:08.993693 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:08.995129 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19321
I20260927 18:37:08.995334 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:08.995879 389 external_mini_cluster.cc:1428] Setting key a0f7bee16709057095e3343ea5076938
I20260927 18:37:08.998829 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--builtin_ntp_servers=127.0.97.84:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:09.006603 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:08.996637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41527 (local address 127.0.97.126:38283)
0927 18:37:08.996840 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:08.996845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:08.998373 (+ 1528us) server_negotiation.cc:407] Connection header received
0927 18:37:09.000249 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.000254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.000338 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.000473 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.001513 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.002322 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.003866 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.004273 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.004403 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.005875 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.005881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.005885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.005903 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.005935 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.005948 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.006025 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.006285 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.006366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:37:09.008154 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:08.997238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38283 (local address 127.0.97.66:41527)
0927 18:37:08.998235 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:08.998269 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:08.999986 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.000697 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.000705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.000793 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:09.001344 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.001360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.002468 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.002481 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.003708 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.003721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.004507 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.004516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.004734 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.005448 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:09.005454 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:09.005508 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:09.005611 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:09.006053 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:09.006064 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:09.006074 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.006281 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.006701 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":309,"threads_started":1}
I20260927 18:37:09.009647 19448 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38283
I20260927 18:37:09.010002 19448 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:09.010823 19448 heartbeater.cc:511] Master 127.0.97.126:38283 requested a full tablet report, sending...
I20260927 18:37:09.012276 19131 ts_manager.cc:194] Registered new tserver with Master: 13b08c5d97e74cc4ae3fd7a4191611c7 (127.0.97.66:33767)
I20260927 18:37:09.013064 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:41527
W20260927 18:37:09.330045 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:09.330363 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:09.330518 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:09.338061 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:09.338188 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:09.338361 19451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:37:09.347987 19451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:09.349318 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:09.351082 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:09.366274 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:09.366513 19457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:09.366791 19456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:09.366909 19451 server_base.cc:1034] running on GCE node
I20260927 18:37:09.367547 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:09.368379 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:09.369571 19451 hybrid_clock.cc:648] HybridClock initialized: now 1790534229369538 us; error 30 us; skew 500 ppm
I20260927 18:37:09.369944 19451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:09.372175 19451 webserver.cc:460] Webserver started at http://127.0.97.67:44131/ using document root <none> and password file <none>
I20260927 18:37:09.372874 19451 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:09.372982 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:09.373328 19451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:09.376202 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391"
format_stamp: "Formatted at 2026-09-27 18:37:09 on dist-test-slave-n3p4"
server_key: "7465d20f727f40d9b120ac14f0134db4"
server_key_iv: "c2de9792fe9f71f6f937c62be817c7e4"
server_key_version: "encryptionkey@0"
I20260927 18:37:09.376958 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391"
format_stamp: "Formatted at 2026-09-27 18:37:09 on dist-test-slave-n3p4"
server_key: "7465d20f727f40d9b120ac14f0134db4"
server_key_iv: "c2de9792fe9f71f6f937c62be817c7e4"
server_key_version: "encryptionkey@0"
I20260927 18:37:09.381757 19451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:37:09.385016 19465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:09.386427 19451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:09.386595 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391"
format_stamp: "Formatted at 2026-09-27 18:37:09 on dist-test-slave-n3p4"
server_key: "7465d20f727f40d9b120ac14f0134db4"
server_key_iv: "c2de9792fe9f71f6f937c62be817c7e4"
server_key_version: "encryptionkey@0"
I20260927 18:37:09.386804 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:09.415333 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:09.416200 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:09.416510 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:09.417501 19451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:09.418949 19451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:09.419060 19451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:09.419189 19451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:09.419291 19451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:09.454332 19451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40235
I20260927 18:37:09.454423 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40235 every 8 connection(s)
I20260927 18:37:09.455651 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:09.464536 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19451
I20260927 18:37:09.464727 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:09.465143 389 external_mini_cluster.cc:1428] Setting key 5e4ff82558556af39b0a863eda39679e
I20260927 18:37:09.469638 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.458514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45617 (local address 127.0.97.126:38283)
0927 18:37:09.458702 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.458706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.460256 (+ 1550us) server_negotiation.cc:407] Connection header received
0927 18:37:09.462170 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.462175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.462275 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.462385 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.463443 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.464316 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.466689 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.467261 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.467355 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.468966 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.468970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.468974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.468990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.469021 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.469032 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.469102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.469316 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.469391 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:37:09.470647 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.459101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38283 (local address 127.0.97.67:45617)
0927 18:37:09.460085 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.460121 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.461911 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.462611 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.462620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.462718 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:09.463271 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.463288 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.464538 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.464543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.466548 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.466561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.467520 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.467530 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.467844 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.468662 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:09.468665 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:09.468709 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:09.468797 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:09.469118 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:09.469125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:09.469131 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.469292 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.469580 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":289,"threads_started":1}
I20260927 18:37:09.472003 19578 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38283
I20260927 18:37:09.472322 19578 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:09.473199 19578 heartbeater.cc:511] Master 127.0.97.126:38283 requested a full tablet report, sending...
I20260927 18:37:09.474593 19131 ts_manager.cc:194] Registered new tserver with Master: 1f8ad9972e1a4873a3b0bdbaa71cf391 (127.0.97.67:40235)
I20260927 18:37:09.475364 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:45617
I20260927 18:37:09.480554 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:09.491539 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.484862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.97.126:38283)
0927 18:37:09.485048 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.485056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.485489 (+ 433us) server_negotiation.cc:407] Connection header received
0927 18:37:09.485687 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.485693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.485774 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.485932 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.486467 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.487257 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.488784 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.489253 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.489399 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.490765 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.490772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.490777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.490805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.490835 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.490848 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.491020 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.491225 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.491284 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:09.495375 19131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:09.502135 19131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:09.552812 19317 heartbeater.cc:503] Master 127.0.97.126:38283 was elected leader, sending a full tablet report...
I20260927 18:37:09.553728 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.516172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.97.65:46861)
0927 18:37:09.541180 (+ 25008us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.541194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.543918 (+ 2724us) server_negotiation.cc:407] Connection header received
0927 18:37:09.544132 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.544142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.544388 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.544594 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:09.546197 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.547470 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.550384 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.551661 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.551791 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.552096 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.553156 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.553352 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24733,"thread_start_us":24570,"threads_started":1}
I20260927 18:37:09.554327 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.515525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35012)
0927 18:37:09.543747 (+ 28222us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.543781 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.543960 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.545128 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.545133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.545169 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:09.545961 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.545975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.547654 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.547660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.550179 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.550191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.552216 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.552222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.552453 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.552489 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.552638 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.552846 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28041,"mutex_wait_us":300,"thread_start_us":27894,"threads_started":1}
I20260927 18:37:09.554647 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.516645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.0.97.67:40235)
0927 18:37:09.538885 (+ 22240us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.538901 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.544232 (+ 5331us) server_negotiation.cc:407] Connection header received
0927 18:37:09.544565 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.544572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.544895 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.545087 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:09.546204 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.547769 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.551310 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.552883 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.553008 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.553299 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.554049 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.554235 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21978,"thread_start_us":22263,"threads_started":1}
I20260927 18:37:09.554486 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.515285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33767 (local address 127.0.0.1:58456)
0927 18:37:09.539914 (+ 24629us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.539980 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.540251 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.544715 (+ 4464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.544724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.544758 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:09.545333 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.545352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.548059 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.548064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.551001 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.551018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.553423 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.553432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.553749 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.553858 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.553956 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.554101 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24256,"mutex_wait_us":275,"thread_start_us":28485,"threads_started":1}
I20260927 18:37:09.555509 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.515903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40235 (local address 127.0.0.1:44696)
0927 18:37:09.544096 (+ 28193us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.544121 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.544349 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.545268 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.545282 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.545309 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:09.545976 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.545985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.547981 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.547986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.551086 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.551097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.553475 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.553481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.553765 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.553861 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.554929 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.555067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28058,"mutex_wait_us":322,"thread_start_us":25321,"threads_started":1}
I20260927 18:37:09.555523 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.515473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.0.97.66:33767)
0927 18:37:09.542650 (+ 27177us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.542663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.542709 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:37:09.542851 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.542864 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.544178 (+ 1314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.544392 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:09.545567 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.547893 (+ 2326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.551281 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.552886 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.553024 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.553263 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.554614 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.554816 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26930,"thread_start_us":28046,"threads_started":1}
I20260927 18:37:09.559310 19252 tablet_service.cc:1467] Processing CreateTablet for tablet 034da3ea873143708cced5d5178f091c (DEFAULT_TABLE table=test-table [id=4af07e8e5c334b0cbeaf48f73314e5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:09.559310 19383 tablet_service.cc:1467] Processing CreateTablet for tablet 034da3ea873143708cced5d5178f091c (DEFAULT_TABLE table=test-table [id=4af07e8e5c334b0cbeaf48f73314e5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:09.559322 19513 tablet_service.cc:1467] Processing CreateTablet for tablet 034da3ea873143708cced5d5178f091c (DEFAULT_TABLE table=test-table [id=4af07e8e5c334b0cbeaf48f73314e5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:09.560746 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034da3ea873143708cced5d5178f091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:09.560760 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034da3ea873143708cced5d5178f091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:09.560747 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034da3ea873143708cced5d5178f091c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:09.572906 19596 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Bootstrap starting.
I20260927 18:37:09.576110 19598 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: Bootstrap starting.
I20260927 18:37:09.576905 19596 tablet_bootstrap.cc:654] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:09.578308 19596 log.cc:824] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:09.579572 19598 tablet_bootstrap.cc:654] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:09.580785 19598 log.cc:824] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:09.581650 19597 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Bootstrap starting.
I20260927 18:37:09.581765 19596 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: No bootstrap required, opened a new log
I20260927 18:37:09.582046 19596 ts_tablet_manager.cc:1397] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:37:09.584538 19598 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: No bootstrap required, opened a new log
I20260927 18:37:09.584933 19598 ts_tablet_manager.cc:1397] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 18:37:09.585335 19597 tablet_bootstrap.cc:654] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:09.586303 19597 log.cc:824] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:09.588671 19597 tablet_bootstrap.cc:492] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: No bootstrap required, opened a new log
I20260927 18:37:09.588889 19597 ts_tablet_manager.cc:1397] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:37:09.592788 19596 raft_consensus.cc:348] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.593286 19596 raft_consensus.cc:374] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:09.593408 19596 raft_consensus.cc:729] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8ad9972e1a4873a3b0bdbaa71cf391, State: Initialized, Role: FOLLOWER
I20260927 18:37:09.594246 19596 consensus_queue.cc:260] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.595556 19578 heartbeater.cc:503] Master 127.0.97.126:38283 was elected leader, sending a full tablet report...
I20260927 18:37:09.596228 19598 raft_consensus.cc:348] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.596639 19598 raft_consensus.cc:374] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:09.596763 19598 raft_consensus.cc:729] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0efa8306dda427897127b59df2abb23, State: Initialized, Role: FOLLOWER
I20260927 18:37:09.596659 19597 raft_consensus.cc:348] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.597077 19597 raft_consensus.cc:374] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:09.597173 19597 raft_consensus.cc:729] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b08c5d97e74cc4ae3fd7a4191611c7, State: Initialized, Role: FOLLOWER
I20260927 18:37:09.597599 19598 consensus_queue.cc:260] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.597860 19596 ts_tablet_manager.cc:1428] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
I20260927 18:37:09.597750 19597 consensus_queue.cc:260] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.599676 19448 heartbeater.cc:503] Master 127.0.97.126:38283 was elected leader, sending a full tablet report...
I20260927 18:37:09.600123 19597 ts_tablet_manager.cc:1428] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260927 18:37:09.600144 19598 ts_tablet_manager.cc:1428] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 18:37:09.617585 19603 raft_consensus.cc:482] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:09.617972 19603 raft_consensus.cc:504] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.620206 19603 leader_election.cc:288] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b08c5d97e74cc4ae3fd7a4191611c7 (127.0.97.66:33767), 1f8ad9972e1a4873a3b0bdbaa71cf391 (127.0.97.67:40235)
I20260927 18:37:09.627210 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.620522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44259 (local address 127.0.97.66:33767)
0927 18:37:09.620708 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.620720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.621341 (+ 621us) server_negotiation.cc:407] Connection header received
0927 18:37:09.621547 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.621553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.621633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.621730 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:09.622265 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.623021 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.625200 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.626155 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.626287 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.626382 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.626837 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.626963 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:37:09.627269 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.620510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33767 (local address 127.0.97.65:44259)
0927 18:37:09.621241 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.621261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.621402 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.621744 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.621748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.621771 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:09.622135 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.622143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.623181 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.623187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.625024 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.625036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.626509 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.626515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.626642 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.626704 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.626793 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.626927 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":5439,"threads_started":1}
I20260927 18:37:09.628216 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034da3ea873143708cced5d5178f091c" candidate_uuid: "b0efa8306dda427897127b59df2abb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" is_pre_election: true
I20260927 18:37:09.628733 19403 raft_consensus.cc:2454] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0efa8306dda427897127b59df2abb23 in term 0.
I20260927 18:37:09.629365 19208 leader_election.cc:302] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b08c5d97e74cc4ae3fd7a4191611c7, b0efa8306dda427897127b59df2abb23; no voters:
I20260927 18:37:09.629741 19603 raft_consensus.cc:2781] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:09.629935 19603 raft_consensus.cc:482] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:09.630050 19603 raft_consensus.cc:3037] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:09.633402 19603 raft_consensus.cc:504] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.634135 19603 leader_election.cc:288] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [CANDIDATE]: Term 1 election: Requested vote from peers 13b08c5d97e74cc4ae3fd7a4191611c7 (127.0.97.66:33767), 1f8ad9972e1a4873a3b0bdbaa71cf391 (127.0.97.67:40235)
I20260927 18:37:09.634186 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.621082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48321 (local address 127.0.97.67:40235)
0927 18:37:09.621257 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.621264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.626336 (+ 5072us) server_negotiation.cc:407] Connection header received
0927 18:37:09.626497 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.626503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.626591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.626709 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:09.627729 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.628794 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.631490 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.632821 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.632939 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.633061 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.633716 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.633889 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:37:09.634586 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034da3ea873143708cced5d5178f091c" candidate_uuid: "b0efa8306dda427897127b59df2abb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7"
I20260927 18:37:09.634778 19403 raft_consensus.cc:3037] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:09.634912 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.620530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40235 (local address 127.0.97.65:48321)
0927 18:37:09.626227 (+ 5697us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.626245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:09.626364 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:09.626888 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:09.626896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:09.626928 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:09.627329 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.627509 (+ 180us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.628988 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.628995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.631273 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:09.631291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.633252 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.633261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:09.633477 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.633535 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 18:37:09.634369 (+ 834us) client_negotiation.cc:240] Negotiation successful
0927 18:37:09.634540 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5578,"thread_start_us":392,"threads_started":1}
I20260927 18:37:09.635927 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034da3ea873143708cced5d5178f091c" candidate_uuid: "b0efa8306dda427897127b59df2abb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391"
I20260927 18:37:09.636008 19533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034da3ea873143708cced5d5178f091c" candidate_uuid: "b0efa8306dda427897127b59df2abb23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" is_pre_election: true
I20260927 18:37:09.636402 19532 raft_consensus.cc:3037] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:09.637905 19403 raft_consensus.cc:2454] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0efa8306dda427897127b59df2abb23 in term 1.
I20260927 18:37:09.638388 19208 leader_election.cc:302] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b08c5d97e74cc4ae3fd7a4191611c7, b0efa8306dda427897127b59df2abb23; no voters:
I20260927 18:37:09.638662 19603 raft_consensus.cc:2781] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:09.639168 19603 raft_consensus.cc:686] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 1 LEADER]: Becoming Leader. State: Replica: b0efa8306dda427897127b59df2abb23, State: Running, Role: LEADER
I20260927 18:37:09.639760 19603 consensus_queue.cc:237] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } }
I20260927 18:37:09.640421 19532 raft_consensus.cc:2454] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0efa8306dda427897127b59df2abb23 in term 1.
I20260927 18:37:09.644778 19130 catalog_manager.cc:5495] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0efa8306dda427897127b59df2abb23 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "b0efa8306dda427897127b59df2abb23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0efa8306dda427897127b59df2abb23" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 46861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 } health_report { overall_health: UNKNOWN } } }
I20260927 18:37:09.681767 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.675718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.97.65:46861)
0927 18:37:09.675959 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.675966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.675989 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:09.676108 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.676113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.676226 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.676400 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:09.676802 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.677554 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.679127 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.679678 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.679825 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.680586 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:09.681131 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:09.681273 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.681409 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.681471 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:37:09.690063 19532 raft_consensus.cc:1261] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 1 FOLLOWER]: Refusing update from remote peer b0efa8306dda427897127b59df2abb23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:09.690129 19403 raft_consensus.cc:1261] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 1 FOLLOWER]: Refusing update from remote peer b0efa8306dda427897127b59df2abb23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:09.690918 19607 consensus_queue.cc:1035] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8ad9972e1a4873a3b0bdbaa71cf391" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:09.691346 19603 consensus_queue.cc:1035] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b08c5d97e74cc4ae3fd7a4191611c7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:09.716851 19130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:09.717135 19130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:09.720721 19130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ac8fdbc6aa14f7bb91ca0de7d4840f0: Sending DeleteTablet for 3 replicas of tablet 034da3ea873143708cced5d5178f091c
I20260927 18:37:09.721903 19513 tablet_service.cc:1514] Processing DeleteTablet for tablet 034da3ea873143708cced5d5178f091c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:09 UTC) from {username='slave'} at 127.0.0.1:44696
I20260927 18:37:09.721911 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet 034da3ea873143708cced5d5178f091c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:09 UTC) from {username='slave'} at 127.0.0.1:58456
I20260927 18:37:09.721909 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet 034da3ea873143708cced5d5178f091c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:09 UTC) from {username='slave'} at 127.0.0.1:35012
I20260927 18:37:09.724223 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19190
I20260927 18:37:09.734267 19621 tablet_replica.cc:330] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: stopping tablet replica
I20260927 18:37:09.734727 19621 raft_consensus.cc:2229] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:09.735288 19621 raft_consensus.cc:2258] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:09.737335 19622 tablet_replica.cc:330] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: stopping tablet replica
I20260927 18:37:09.737874 19622 raft_consensus.cc:2229] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:09.738494 19622 raft_consensus.cc:2258] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:09.740679 19622 ts_tablet_manager.cc:1905] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:09.741086 19621 ts_tablet_manager.cc:1905] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:09.741475 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.726033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.0.97.65:46861)
0927 18:37:09.726213 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.726219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.730775 (+ 4556us) server_negotiation.cc:407] Connection header received
0927 18:37:09.731797 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.731806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.731939 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.732040 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.733431 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.734699 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.736856 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.737702 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.737862 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.739500 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.739516 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.739538 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.740437 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.740515 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.740547 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.740644 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.740926 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.741034 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:37:09.741739 19623 tablet_replica.cc:330] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23: stopping tablet replica
I20260927 18:37:09.742197 19623 raft_consensus.cc:2229] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:09.742834 19312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:09.749354 19622 ts_tablet_manager.cc:1918] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:09.749540 19622 log.cc:1196] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/034da3ea873143708cced5d5178f091c
I20260927 18:37:09.750089 19622 ts_tablet_manager.cc:1939] T 034da3ea873143708cced5d5178f091c P 1f8ad9972e1a4873a3b0bdbaa71cf391: Deleting consensus metadata
I20260927 18:37:09.751189 19621 ts_tablet_manager.cc:1918] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:09.751350 19621 log.cc:1196] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/034da3ea873143708cced5d5178f091c
I20260927 18:37:09.751715 19117 catalog_manager.cc:4841] TS 1f8ad9972e1a4873a3b0bdbaa71cf391 (127.0.97.67:40235): tablet 034da3ea873143708cced5d5178f091c (table test-table [id=4af07e8e5c334b0cbeaf48f73314e5cf]) successfully deleted
I20260927 18:37:09.751927 19621 ts_tablet_manager.cc:1939] T 034da3ea873143708cced5d5178f091c P 13b08c5d97e74cc4ae3fd7a4191611c7: Deleting consensus metadata
I20260927 18:37:09.753620 19118 catalog_manager.cc:4841] TS 13b08c5d97e74cc4ae3fd7a4191611c7 (127.0.97.66:33767): tablet 034da3ea873143708cced5d5178f091c (table test-table [id=4af07e8e5c334b0cbeaf48f73314e5cf]) successfully deleted
I20260927 18:37:09.742884 19623 raft_consensus.cc:2258] T 034da3ea873143708cced5d5178f091c P b0efa8306dda427897127b59df2abb23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:09.843236 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19190
W20260927 18:37:09.861089 19116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:46861 (error 108)
I20260927 18:37:09.861349 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19321
I20260927 18:37:09.862963 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.862527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35042)
0927 18:37:09.862696 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.862822 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 18:37:09.863206 19116 catalog_manager.cc:4584] TS b0efa8306dda427897127b59df2abb23 (127.0.97.65:46861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 034da3ea873143708cced5d5178f091c: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
I20260927 18:37:09.868104 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.862247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.0.97.66:33767)
0927 18:37:09.862438 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.862444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.862464 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:37:09.862565 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.862571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.862678 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.862805 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.863256 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.864058 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.865308 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.865710 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.865906 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.866904 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.866912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.866927 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.867501 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.867548 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.867567 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.867662 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.867770 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.867855 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:37:09.868757 19443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:09.916786 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.916249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35058)
0927 18:37:09.916485 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.916594 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:37:09.917986 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.917520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35060)
0927 18:37:09.917723 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.917840 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:37:09.958077 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19321
I20260927 18:37:09.977279 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19451
I20260927 18:37:09.978824 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.978232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35068)
0927 18:37:09.978521 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.978628 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 18:37:09.979969 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.979608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35076)
0927 18:37:09.979766 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:09.979849 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:37:09.985872 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:09.978134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.67:40235)
0927 18:37:09.978332 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:09.978338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:09.978493 (+ 155us) server_negotiation.cc:407] Connection header received
0927 18:37:09.978683 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:09.978689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:09.978807 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:09.978923 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:09.979576 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.980752 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.982246 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:09.982915 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:09.983055 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:09.984161 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:09.984183 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:09.984204 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:09.984986 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:09.985049 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:09.985076 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:09.985210 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:09.985356 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:37:09.985472 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:37:09.986649 19573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:10.060699 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:10.060234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35088)
0927 18:37:10.060451 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:10.060539 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:37:10.061849 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:10.061383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:46861 (local address 127.0.0.1:35090)
0927 18:37:10.061595 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:10.061677 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 18:37:10.075234 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19451
I20260927 18:37:10.091776 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19100
I20260927 18:37:10.092868 19161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:10.202769 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19100
2026-09-27T18:37:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2725 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:10 dist-test-slave-n3p4 krb5kdc[19632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:10 dist-test-slave-n3p4 krb5kdc[19632](info): set up 2 sockets
Sep 27 18:37:10 dist-test-slave-n3p4 krb5kdc[19632](info): commencing operation
krb5kdc: starting...
W20260927 18:37:11.302150 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:11 dist-test-slave-n3p4 krb5kdc[19632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:11.553565 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36089
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:11.883715 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:11.884001 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:11.884177 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:11.891539 19645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:11.891654 19645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:11.891772 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:11.891870 19645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:11.891957 19645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:11.902127 19645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:11.903488 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:11.904814 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:11.916581 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:11.916585 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:11.916648 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:11.917867 19645 server_base.cc:1034] running on GCE node
I20260927 18:37:11.918637 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:11.919924 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:11.921131 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790534231921083 us; error 36 us; skew 500 ppm
I20260927 18:37:11.921434 19645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:11 dist-test-slave-n3p4 krb5kdc[19632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:11.930229 19645 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:11.931701 19645 webserver.cc:460] Webserver started at http://127.0.97.126:44419/ using document root <none> and password file <none>
I20260927 18:37:11.932399 19645 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:11.932540 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:11.932907 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:11.935827 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "759d479727104ad393faa0e6a5b4209e"
format_stamp: "Formatted at 2026-09-27 18:37:11 on dist-test-slave-n3p4"
server_key: "4c1a20971b3168842cd67548a8c000b9"
server_key_iv: "ad60c9e3877ea6b2db72bd0becf7fa55"
server_key_version: "encryptionkey@0"
I20260927 18:37:11.936548 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "759d479727104ad393faa0e6a5b4209e"
format_stamp: "Formatted at 2026-09-27 18:37:11 on dist-test-slave-n3p4"
server_key: "4c1a20971b3168842cd67548a8c000b9"
server_key_iv: "ad60c9e3877ea6b2db72bd0becf7fa55"
server_key_version: "encryptionkey@0"
I20260927 18:37:11.941488 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 18:37:11.944826 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:11.946202 19645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:11.946339 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "759d479727104ad393faa0e6a5b4209e"
format_stamp: "Formatted at 2026-09-27 18:37:11 on dist-test-slave-n3p4"
server_key: "4c1a20971b3168842cd67548a8c000b9"
server_key_iv: "ad60c9e3877ea6b2db72bd0becf7fa55"
server_key_version: "encryptionkey@0"
I20260927 18:37:11.946573 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:11.967090 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:11.971436 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:11.971706 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:11.991408 19645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36089
I20260927 18:37:11.991428 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36089 every 8 connection(s)
I20260927 18:37:11.992865 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:11.996592 19712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:11.997661 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19645
I20260927 18:37:11.997881 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:11.998471 389 external_mini_cluster.cc:1428] Setting key 66300abd311b42ae06fc5f6282ea2a93
I20260927 18:37:12.011217 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: Bootstrap starting.
Sep 27 18:37:12 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:12.015059 19712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:12.016093 19712 log.cc:824] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:12.019280 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: No bootstrap required, opened a new log
I20260927 18:37:12.022989 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:12.000521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.0.97.126:36089)
0927 18:37:12.001361 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:12.001372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:12.001413 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:37:12.003161 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:12.003188 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:12.003615 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:12.003971 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:12.005865 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.007163 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.009095 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.009748 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.009946 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:12.014716 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:12.014729 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:12.014749 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:12.014803 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:12.018534 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:12.019195 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:12.019203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:12.019211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:12.019447 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:12.019847 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:12.019852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:12.019855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:12.020508 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:12.020786 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:12.021153 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 18:37:12.021617 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":411,"threads_started":1}
I20260927 18:37:12.028692 19712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } }
I20260927 18:37:12.029016 19712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:12.029086 19712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759d479727104ad393faa0e6a5b4209e, State: Initialized, Role: FOLLOWER
I20260927 18:37:12.029536 19712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } }
I20260927 18:37:12.029724 19712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:12.029892 19712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:12.030022 19712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:12.032169 19712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } }
I20260927 18:37:12.032686 19712 leader_election.cc:302] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759d479727104ad393faa0e6a5b4209e; no voters:
I20260927 18:37:12.033223 19712 leader_election.cc:288] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:12.033381 19717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:12.034057 19717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [term 1 LEADER]: Becoming Leader. State: Replica: 759d479727104ad393faa0e6a5b4209e, State: Running, Role: LEADER
I20260927 18:37:12.034559 19717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } }
I20260927 18:37:12.035204 19712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:12.037209 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759d479727104ad393faa0e6a5b4209e. Latest consensus state: current_term: 1 leader_uuid: "759d479727104ad393faa0e6a5b4209e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } } }
I20260927 18:37:12.037671 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:12.039968 19724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:12.039932 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759d479727104ad393faa0e6a5b4209e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759d479727104ad393faa0e6a5b4209e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36089 } } }
I20260927 18:37:12.040333 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:12.044865 19724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:12.053061 19724 catalog_manager.cc:1347] Generated new cluster ID: 8fbf48c5214d45d6a7a84f3c39857603
I20260927 18:37:12.053161 19724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:12.077152 19724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:12.078724 19724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:12.098974 19724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: Generated new TSK 0
I20260927 18:37:12.099862 19724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260927 18:37:12.161463 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:12.488056 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:12.488343 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:12.488509 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:12.495597 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:12.495741 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:12.495935 19738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:37:12.505445 19738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:12.506769 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:12.508059 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:12.521209 19743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:12.521301 19746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:12.522156 19738 server_base.cc:1034] running on GCE node
W20260927 18:37:12.521252 19744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:12.524084 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:12.524969 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:12.526178 19738 hybrid_clock.cc:648] HybridClock initialized: now 1790534232526110 us; error 53 us; skew 500 ppm
I20260927 18:37:12.526520 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:12 dist-test-slave-n3p4 krb5kdc[19632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:12.529546 19738 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260927 18:37:12.531080 19738 webserver.cc:460] Webserver started at http://127.0.97.65:38953/ using document root <none> and password file <none>
I20260927 18:37:12.531746 19738 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:12.531883 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:12.532179 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:12.535055 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "b1bfcfa159594751a8f28a908aef7fcf"
format_stamp: "Formatted at 2026-09-27 18:37:12 on dist-test-slave-n3p4"
server_key: "8842d4743f586d91a92e79feffed2e45"
server_key_iv: "7b1abf9bbbc1603a6fbe30028339f729"
server_key_version: "encryptionkey@0"
I20260927 18:37:12.535854 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "b1bfcfa159594751a8f28a908aef7fcf"
format_stamp: "Formatted at 2026-09-27 18:37:12 on dist-test-slave-n3p4"
server_key: "8842d4743f586d91a92e79feffed2e45"
server_key_iv: "7b1abf9bbbc1603a6fbe30028339f729"
server_key_version: "encryptionkey@0"
I20260927 18:37:12.540704 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 18:37:12.544289 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:12.545462 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:12.545624 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "b1bfcfa159594751a8f28a908aef7fcf"
format_stamp: "Formatted at 2026-09-27 18:37:12 on dist-test-slave-n3p4"
server_key: "8842d4743f586d91a92e79feffed2e45"
server_key_iv: "7b1abf9bbbc1603a6fbe30028339f729"
server_key_version: "encryptionkey@0"
I20260927 18:37:12.545849 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:12.566648 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:12.570474 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:12.570773 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:12.571704 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:12.572960 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:12.573066 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:12.573213 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:12.573288 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:12.606276 19738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39133
I20260927 18:37:12.606357 19865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39133 every 8 connection(s)
I20260927 18:37:12.607506 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:12.614355 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19738
I20260927 18:37:12.614532 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:12.614995 389 external_mini_cluster.cc:1428] Setting key a268fe5e157247bb830453d4d5c7046f
Sep 27 18:37:12 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:12.627127 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:12.609971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41161 (local address 127.0.97.126:36089)
0927 18:37:12.610400 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:12.610404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:12.611231 (+ 827us) server_negotiation.cc:407] Connection header received
0927 18:37:12.613038 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:12.613044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:12.613139 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:12.613253 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:12.614818 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.615764 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.617736 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.618213 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.618347 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:12.621631 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:12.621638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:12.621644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:12.621681 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:12.624235 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:12.624919 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:12.624925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:12.624932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:12.625054 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:12.625458 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:12.625465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:12.625468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:12.625716 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:12.625892 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:12.626574 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 18:37:12.626786 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":238,"threads_started":1}
I20260927 18:37:12.628044 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:12.610399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36089 (local address 127.0.97.65:41161)
0927 18:37:12.611063 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:12.611106 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:12.612778 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:12.613476 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:12.613484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:12.614015 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:12.614625 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:12.614654 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.615975 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.615979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:12.617585 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:12.617596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:12.618431 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:12.618441 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:12.618686 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:12.619362 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:12.619384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:12.621430 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:12.624375 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:12.624384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:12.624400 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:12.624801 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:12.625162 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:12.625168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:12.625171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:12.625346 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:12.625873 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:12.625881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:12.626206 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 18:37:12.626477 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 18:37:12.626970 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260927 18:37:12.629534 19866 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36089
I20260927 18:37:12.629889 19866 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:12.630702 19866 heartbeater.cc:511] Master 127.0.97.126:36089 requested a full tablet report, sending...
I20260927 18:37:12.633033 19677 ts_manager.cc:194] Registered new tserver with Master: b1bfcfa159594751a8f28a908aef7fcf (127.0.97.65:39133)
I20260927 18:37:12.635245 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:41161
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260927 18:37:12.671142 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:13.002032 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:13.002462 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:13.002712 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:13.010385 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:13.010483 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:13.010593 19872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:37:13.020257 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:13.021417 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:13.022691 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:13.036475 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:13.036484 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:13.036588 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:13.037405 19872 server_base.cc:1034] running on GCE node
I20260927 18:37:13.038010 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:13.039047 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:13.040263 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790534233040201 us; error 60 us; skew 500 ppm
I20260927 18:37:13.040558 19872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:13 dist-test-slave-n3p4 krb5kdc[19632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:13.044130 19872 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260927 18:37:13.045542 19872 webserver.cc:460] Webserver started at http://127.0.97.66:38055/ using document root <none> and password file <none>
I20260927 18:37:13.046273 19872 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:13.046415 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:13.046728 19872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:13.049610 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "f0221dd2758b4dc38f753cfc6a8ec56e"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "9a6ca91881cdf8852bda427bae79e03f"
server_key_iv: "7fe54b0bdb02e325e301a3a1991c1186"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.050484 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f0221dd2758b4dc38f753cfc6a8ec56e"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "9a6ca91881cdf8852bda427bae79e03f"
server_key_iv: "7fe54b0bdb02e325e301a3a1991c1186"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.055354 19872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:37:13.058703 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.059995 19872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:37:13.060166 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "f0221dd2758b4dc38f753cfc6a8ec56e"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "9a6ca91881cdf8852bda427bae79e03f"
server_key_iv: "7fe54b0bdb02e325e301a3a1991c1186"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.060384 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:13.072232 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:13.076030 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:13.076314 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:13.077220 19872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:13.078548 19872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:13.078650 19872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.078799 19872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:13.078872 19872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.112396 19872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:40031
I20260927 18:37:13.112483 19999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:40031 every 8 connection(s)
I20260927 18:37:13.113669 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:13.114571 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 19872
I20260927 18:37:13.114746 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:13.115295 389 external_mini_cluster.cc:1428] Setting key b0468332abe7d2af01f068518453ca15
Sep 27 18:37:13 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:13.134877 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.116607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34653 (local address 127.0.97.126:36089)
0927 18:37:13.116804 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.116812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.118121 (+ 1309us) server_negotiation.cc:407] Connection header received
0927 18:37:13.119891 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.119897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.119986 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.120093 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:13.121565 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.122333 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.124421 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.124866 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.124999 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.129107 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:13.129122 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:13.129126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:13.129159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:13.131730 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.132488 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.132496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.132499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.132641 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.133066 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.133075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.133078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.133355 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:13.133523 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.134276 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.134513 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:37:13.136047 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.117199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36089 (local address 127.0.97.66:34653)
0927 18:37:13.117980 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.118018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.119640 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.120336 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.120344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.120843 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:13.121396 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.121414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.122527 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.122534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.124267 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.124283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.125079 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.125086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.125328 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.126044 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:13.126066 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:13.128851 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:13.131904 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:13.131913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:13.131929 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:13.132369 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:13.132752 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:13.132757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:13.132761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:13.132946 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:13.133492 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:13.133501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:13.133918 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.134186 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.134728 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":308,"threads_started":1}
I20260927 18:37:13.137648 20000 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36089
I20260927 18:37:13.138036 20000 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:13.138926 20000 heartbeater.cc:511] Master 127.0.97.126:36089 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260927 18:37:13.140604 19677 ts_manager.cc:194] Registered new tserver with Master: f0221dd2758b4dc38f753cfc6a8ec56e (127.0.97.66:40031)
I20260927 18:37:13.141525 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:34653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260927 18:37:13.172979 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:13.500558 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:13.500839 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:13.501026 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:13.508268 20005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:13.508381 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:13.508610 20005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:37:13.518805 20005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:13.520079 20005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:13.521468 20005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:13.534950 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:13.535041 20013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:13.535018 20011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:13.536191 20005 server_base.cc:1034] running on GCE node
I20260927 18:37:13.536862 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:13.537897 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:13.539122 20005 hybrid_clock.cc:648] HybridClock initialized: now 1790534233539049 us; error 60 us; skew 500 ppm
I20260927 18:37:13.539441 20005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:13 dist-test-slave-n3p4 krb5kdc[19632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:13.542570 20005 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260927 18:37:13.544101 20005 webserver.cc:460] Webserver started at http://127.0.97.67:35333/ using document root <none> and password file <none>
I20260927 18:37:13.544775 20005 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:13.544917 20005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:13.545228 20005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:13.548136 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "7894d22b008a434baf8de4e4e8ae06a7"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "83f0861d413813219f3d3a8aac42346a"
server_key_iv: "e3a913d75c3dc08e71265d73b3a66ce8"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.548914 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "7894d22b008a434baf8de4e4e8ae06a7"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "83f0861d413813219f3d3a8aac42346a"
server_key_iv: "e3a913d75c3dc08e71265d73b3a66ce8"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.554327 20005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 18:37:13.557904 20020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.559312 20005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:13.559477 20005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "7894d22b008a434baf8de4e4e8ae06a7"
format_stamp: "Formatted at 2026-09-27 18:37:13 on dist-test-slave-n3p4"
server_key: "83f0861d413813219f3d3a8aac42346a"
server_key_iv: "e3a913d75c3dc08e71265d73b3a66ce8"
server_key_version: "encryptionkey@0"
I20260927 18:37:13.559682 20005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:13.600783 20005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:13.604790 20005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:13.605089 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:13.606287 20005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:13.608009 20005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:13.608108 20005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.608219 20005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:13.608287 20005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:13.639156 19866 heartbeater.cc:503] Master 127.0.97.126:36089 was elected leader, sending a full tablet report...
I20260927 18:37:13.643790 20005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43785
I20260927 18:37:13.643874 20132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43785 every 8 connection(s)
I20260927 18:37:13.645057 20005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:13.645339 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20005
I20260927 18:37:13.645501 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:13.646116 389 external_mini_cluster.cc:1428] Setting key a9daac376b12390bb51710a086681e40
Sep 27 18:37:13 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:13.665496 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.647835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40117 (local address 127.0.97.126:36089)
0927 18:37:13.648293 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.648297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.649512 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 18:37:13.651615 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.651621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.651701 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.651824 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:13.653231 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.654099 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.655607 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.656011 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.656137 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.659531 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:13.659537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:13.659541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:13.659582 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:13.662441 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.663215 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.663222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.663229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.663356 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.663782 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.663787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.663791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.664118 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:13.664266 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.664909 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.665120 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":256,"threads_started":1}
I20260927 18:37:13.666468 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.648500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36089 (local address 127.0.97.67:40117)
0927 18:37:13.649355 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.649411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.651369 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.652049 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.652057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.652531 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:13.653063 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.653080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.654249 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.654254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.655455 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.655466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.656216 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.656223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.656448 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.657083 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:13.657104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:13.659300 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:13.662575 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:13.662583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:13.662597 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:13.663101 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:13.663464 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:13.663469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:13.663472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:13.663673 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:13.664252 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:13.664260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:13.664569 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.664822 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.665351 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":333,"threads_started":1}
I20260927 18:37:13.667642 20133 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36089
I20260927 18:37:13.667941 20133 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:13.668632 20133 heartbeater.cc:511] Master 127.0.97.126:36089 requested a full tablet report, sending...
I20260927 18:37:13.670003 19677 ts_manager.cc:194] Registered new tserver with Master: 7894d22b008a434baf8de4e4e8ae06a7 (127.0.97.67:43785)
I20260927 18:37:13.670784 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:40117
I20260927 18:37:13.673167 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:13.689421 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.677235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.0.97.126:36089)
0927 18:37:13.677392 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.677396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.677942 (+ 546us) server_negotiation.cc:407] Connection header received
0927 18:37:13.678077 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.678080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.678134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.678212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:13.679529 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.680174 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.681304 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.681668 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.681751 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.684100 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:13.684112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:13.684115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:13.684145 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:13.686917 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.687539 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.687543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.687545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.687660 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.688051 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.688055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.688057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.688412 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:13.688607 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.688993 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.689182 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:37:13.693377 19677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:13.700085 19677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:13.722963 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.711924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39133 (local address 127.0.0.1:56986)
0927 18:37:13.712604 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.712645 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.712806 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.714480 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.714489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.714528 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:13.715024 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.715038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.717299 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.717307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.719935 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.719955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.721878 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.721886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.722126 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.722206 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.722311 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.722495 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":31,"thread_start_us":286,"threads_started":1}
I20260927 18:37:13.723865 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.712294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.0.97.65:39133)
0927 18:37:13.713847 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.713858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.713887 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:37:13.713995 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.714002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.714243 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.715277 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:13.715496 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.717145 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.720133 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.721631 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.723052 (+ 1421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.723277 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.723368 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.723551 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":1204,"threads_started":1}
I20260927 18:37:13.724284 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.712250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40031 (local address 127.0.0.1:59166)
0927 18:37:13.713326 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.713345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.713471 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.714672 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.714676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.714694 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:13.715112 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.715124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.717909 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.717914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.720335 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.720348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.722139 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.722144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.722655 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.722689 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.722771 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.722914 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":387,"thread_start_us":1022,"threads_started":1}
I20260927 18:37:13.724632 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.712976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.0.97.66:40031)
0927 18:37:13.714119 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.714127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.714150 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:13.714231 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.714237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.714468 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.715277 (+ 809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:13.715496 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.717745 (+ 2249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.720500 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.721917 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.722212 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.722419 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.724109 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.724310 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":985,"threads_started":1}
I20260927 18:37:13.728266 19801 tablet_service.cc:1467] Processing CreateTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 (DEFAULT_TABLE table=test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:13.729316 19935 tablet_service.cc:1467] Processing CreateTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 (DEFAULT_TABLE table=test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:13.730000 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9fe4007820f49aca66f96cf8e10b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:13.730806 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9fe4007820f49aca66f96cf8e10b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:13.732306 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.713838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43785 (local address 127.0.0.1:52878)
0927 18:37:13.723185 (+ 9347us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.723203 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.723301 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.726487 (+ 3186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.726492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.726517 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:13.726688 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.726698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.728198 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.728206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.730175 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.730191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.731534 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.731540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.731722 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.731754 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.731877 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.732019 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9261,"thread_start_us":11285,"threads_started":1}
I20260927 18:37:13.732419 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.714304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.0.97.67:43785)
0927 18:37:13.725872 (+ 11568us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.725894 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.725949 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:37:13.726043 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.726053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.726296 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.726507 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:13.727048 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.728058 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.730358 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.731379 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.731523 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.731748 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.731914 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.732074 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11356,"thread_start_us":11187,"threads_started":1}
I20260927 18:37:13.735759 20068 tablet_service.cc:1467] Processing CreateTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 (DEFAULT_TABLE table=test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:13.736927 20068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9fe4007820f49aca66f96cf8e10b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:13.746356 20153 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Bootstrap starting.
I20260927 18:37:13.749140 20152 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Bootstrap starting.
I20260927 18:37:13.750599 20154 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Bootstrap starting.
I20260927 18:37:13.750973 20153 tablet_bootstrap.cc:654] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:13.752368 20153 log.cc:824] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:13.753106 20152 tablet_bootstrap.cc:654] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:13.754331 20154 tablet_bootstrap.cc:654] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:13.754485 20152 log.cc:824] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:13.755332 20153 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: No bootstrap required, opened a new log
I20260927 18:37:13.755584 20154 log.cc:824] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:13.755606 20153 ts_tablet_manager.cc:1397] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:37:13.757386 20152 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: No bootstrap required, opened a new log
I20260927 18:37:13.757654 20152 ts_tablet_manager.cc:1397] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 18:37:13.758550 20154 tablet_bootstrap.cc:492] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: No bootstrap required, opened a new log
I20260927 18:37:13.758795 20154 ts_tablet_manager.cc:1397] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 18:37:13.763867 20153 raft_consensus.cc:348] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.764519 20153 raft_consensus.cc:374] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:13.764683 20153 raft_consensus.cc:729] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1bfcfa159594751a8f28a908aef7fcf, State: Initialized, Role: FOLLOWER
I20260927 18:37:13.765319 20153 consensus_queue.cc:260] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.767036 20153 ts_tablet_manager.cc:1428] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Time spent starting tablet: real 0.011s user 0.002s sys 0.010s
I20260927 18:37:13.769065 20154 raft_consensus.cc:348] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.769121 20152 raft_consensus.cc:348] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.769436 20154 raft_consensus.cc:374] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:13.769555 20154 raft_consensus.cc:729] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7894d22b008a434baf8de4e4e8ae06a7, State: Initialized, Role: FOLLOWER
I20260927 18:37:13.769584 20152 raft_consensus.cc:374] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:13.769673 20152 raft_consensus.cc:729] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0221dd2758b4dc38f753cfc6a8ec56e, State: Initialized, Role: FOLLOWER
I20260927 18:37:13.770188 20154 consensus_queue.cc:260] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.770285 20152 consensus_queue.cc:260] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.772387 20152 ts_tablet_manager.cc:1428] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 18:37:13.773198 20133 heartbeater.cc:503] Master 127.0.97.126:36089 was elected leader, sending a full tablet report...
I20260927 18:37:13.773454 20000 heartbeater.cc:503] Master 127.0.97.126:36089 was elected leader, sending a full tablet report...
I20260927 18:37:13.774571 20154 ts_tablet_manager.cc:1428] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 18:37:13.774561 20159 raft_consensus.cc:482] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:13.774928 20159 raft_consensus.cc:504] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.777967 20159 leader_election.cc:288] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1bfcfa159594751a8f28a908aef7fcf (127.0.97.65:39133), f0221dd2758b4dc38f753cfc6a8ec56e (127.0.97.66:40031)
I20260927 18:37:13.783916 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.778281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52687 (local address 127.0.97.66:40031)
0927 18:37:13.778578 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.778586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.778642 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:37:13.778752 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.778760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.778932 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.779105 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:13.779562 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.780606 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.782232 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.783011 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.783161 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.783267 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.783544 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.783661 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:37:13.783911 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.777728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40031 (local address 127.0.97.67:52687)
0927 18:37:13.777977 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.777999 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.778109 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.779165 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.779169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.779198 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:13.779409 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.779417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.780733 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.780745 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.782083 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.782097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.783202 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.783208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.783389 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.783412 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.783511 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.783641 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 18:37:13.785007 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.778307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39133 (local address 127.0.97.67:56867)
0927 18:37:13.779815 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:13.779833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:13.780003 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:13.780476 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:13.780481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:13.780504 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:13.780837 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.780849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.781824 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.781830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.783130 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:13.783140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.784337 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.784343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:13.784516 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.784546 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:37:13.784617 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:37:13.784695 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1156,"threads_started":1}
I20260927 18:37:13.785207 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.777745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56867 (local address 127.0.97.65:39133)
0927 18:37:13.778088 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.778094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.779906 (+ 1812us) server_negotiation.cc:407] Connection header received
0927 18:37:13.780132 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.780137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.780207 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.780317 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:13.780990 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.781663 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.783292 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.784142 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.784279 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.784357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.784775 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.784930 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 18:37:13.785000 19955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9fe4007820f49aca66f96cf8e10b9a8" candidate_uuid: "7894d22b008a434baf8de4e4e8ae06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" is_pre_election: true
I20260927 18:37:13.785540 19955 raft_consensus.cc:2454] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7894d22b008a434baf8de4e4e8ae06a7 in term 0.
I20260927 18:37:13.786294 20024 leader_election.cc:302] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7894d22b008a434baf8de4e4e8ae06a7, f0221dd2758b4dc38f753cfc6a8ec56e; no voters:
I20260927 18:37:13.786748 20159 raft_consensus.cc:2781] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:13.786697 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9fe4007820f49aca66f96cf8e10b9a8" candidate_uuid: "7894d22b008a434baf8de4e4e8ae06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1bfcfa159594751a8f28a908aef7fcf" is_pre_election: true
I20260927 18:37:13.786914 20159 raft_consensus.cc:482] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:13.787029 20159 raft_consensus.cc:3037] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:13.787202 19821 raft_consensus.cc:2454] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7894d22b008a434baf8de4e4e8ae06a7 in term 0.
I20260927 18:37:13.789386 20159 raft_consensus.cc:504] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.790083 20159 leader_election.cc:288] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [CANDIDATE]: Term 1 election: Requested vote from peers b1bfcfa159594751a8f28a908aef7fcf (127.0.97.65:39133), f0221dd2758b4dc38f753cfc6a8ec56e (127.0.97.66:40031)
I20260927 18:37:13.790549 19955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9fe4007820f49aca66f96cf8e10b9a8" candidate_uuid: "7894d22b008a434baf8de4e4e8ae06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e"
I20260927 18:37:13.790549 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9fe4007820f49aca66f96cf8e10b9a8" candidate_uuid: "7894d22b008a434baf8de4e4e8ae06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1bfcfa159594751a8f28a908aef7fcf"
I20260927 18:37:13.790766 19955 raft_consensus.cc:3037] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:13.790818 19821 raft_consensus.cc:3037] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:13.794106 19821 raft_consensus.cc:2454] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7894d22b008a434baf8de4e4e8ae06a7 in term 1.
I20260927 18:37:13.794100 19955 raft_consensus.cc:2454] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7894d22b008a434baf8de4e4e8ae06a7 in term 1.
I20260927 18:37:13.794539 20022 leader_election.cc:302] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7894d22b008a434baf8de4e4e8ae06a7, b1bfcfa159594751a8f28a908aef7fcf; no voters:
I20260927 18:37:13.794937 20159 raft_consensus.cc:2781] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:13.795490 20159 raft_consensus.cc:686] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7894d22b008a434baf8de4e4e8ae06a7, State: Running, Role: LEADER
I20260927 18:37:13.796012 20159 consensus_queue.cc:237] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } }
I20260927 18:37:13.800309 19677 catalog_manager.cc:5495] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7894d22b008a434baf8de4e4e8ae06a7 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "7894d22b008a434baf8de4e4e8ae06a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7894d22b008a434baf8de4e4e8ae06a7" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260927 18:37:13.834054 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.827734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.97.67:43785)
0927 18:37:13.827980 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.827988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.828110 (+ 122us) server_negotiation.cc:407] Connection header received
0927 18:37:13.828381 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.828385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.828479 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.828650 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:13.829069 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.829913 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.831685 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.832107 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.832273 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.832825 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:13.833363 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:13.833523 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.833650 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.833741 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:37:13.842321 19821 raft_consensus.cc:1261] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 1 FOLLOWER]: Refusing update from remote peer 7894d22b008a434baf8de4e4e8ae06a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:13.842492 19955 raft_consensus.cc:1261] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 1 FOLLOWER]: Refusing update from remote peer 7894d22b008a434baf8de4e4e8ae06a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:13.843397 20162 consensus_queue.cc:1035] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1bfcfa159594751a8f28a908aef7fcf" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:13.843875 20159 consensus_queue.cc:1035] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0221dd2758b4dc38f753cfc6a8ec56e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:13.872478 19677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:13.872704 19677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:13.876530 19677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 759d479727104ad393faa0e6a5b4209e: Sending DeleteTablet for 3 replicas of tablet e9fe4007820f49aca66f96cf8e10b9a8
I20260927 18:37:13.877552 19801 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:13 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56986
I20260927 18:37:13.877743 20068 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:13 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52878
I20260927 18:37:13.878000 19935 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fe4007820f49aca66f96cf8e10b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:13 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:59166
I20260927 18:37:13.880061 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19738
I20260927 18:37:13.889175 20177 tablet_replica.cc:330] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: stopping tablet replica
I20260927 18:37:13.889740 20177 raft_consensus.cc:2229] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:13.890374 20177 raft_consensus.cc:2258] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:13.892714 20176 tablet_replica.cc:330] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: stopping tablet replica
I20260927 18:37:13.892848 20175 tablet_replica.cc:330] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: stopping tablet replica
I20260927 18:37:13.893119 20176 raft_consensus.cc:2229] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:13.893198 20175 raft_consensus.cc:2229] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:13.893654 20175 raft_consensus.cc:2258] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:13.893769 20176 raft_consensus.cc:2258] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:13.895818 20177 ts_tablet_manager.cc:1905] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:13.895835 20175 ts_tablet_manager.cc:1905] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:37:13 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260927 18:37:13.902788 20177 ts_tablet_manager.cc:1918] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:13.902801 20175 ts_tablet_manager.cc:1918] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:13.902957 20175 log.cc:1196] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/e9fe4007820f49aca66f96cf8e10b9a8
I20260927 18:37:13.902957 20177 log.cc:1196] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/e9fe4007820f49aca66f96cf8e10b9a8
I20260927 18:37:13.903497 20177 ts_tablet_manager.cc:1939] T e9fe4007820f49aca66f96cf8e10b9a8 P f0221dd2758b4dc38f753cfc6a8ec56e: Deleting consensus metadata
I20260927 18:37:13.903499 20175 ts_tablet_manager.cc:1939] T e9fe4007820f49aca66f96cf8e10b9a8 P b1bfcfa159594751a8f28a908aef7fcf: Deleting consensus metadata
I20260927 18:37:13.905342 19664 catalog_manager.cc:4841] TS f0221dd2758b4dc38f753cfc6a8ec56e (127.0.97.66:40031): tablet e9fe4007820f49aca66f96cf8e10b9a8 (table test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]) successfully deleted
I20260927 18:37:13.905695 19662 catalog_manager.cc:4841] TS b1bfcfa159594751a8f28a908aef7fcf (127.0.97.65:39133): tablet e9fe4007820f49aca66f96cf8e10b9a8 (table test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]) successfully deleted
I20260927 18:37:13.907351 20176 ts_tablet_manager.cc:1905] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:13.909621 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:13.881683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.97.65:39133)
0927 18:37:13.881915 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:13.881923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:13.886093 (+ 4170us) server_negotiation.cc:407] Connection header received
0927 18:37:13.886315 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:13.886321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:13.886409 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:13.886550 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:13.890915 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.892163 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.894973 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:13.895630 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:13.895750 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:13.901951 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:13.901966 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:13.901985 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:13.902036 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:13.906220 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.907193 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.907205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.907211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.907365 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:13.907920 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:13.907928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:13.907934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:13.908378 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:13.908665 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:13.909037 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 18:37:13.909259 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:37:13.910651 19861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:13.913209 20176 ts_tablet_manager.cc:1918] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:13.913409 20176 log.cc:1196] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/e9fe4007820f49aca66f96cf8e10b9a8
I20260927 18:37:13.913998 20176 ts_tablet_manager.cc:1939] T e9fe4007820f49aca66f96cf8e10b9a8 P 7894d22b008a434baf8de4e4e8ae06a7: Deleting consensus metadata
I20260927 18:37:13.915886 19664 catalog_manager.cc:4841] TS 7894d22b008a434baf8de4e4e8ae06a7 (127.0.97.67:43785): tablet e9fe4007820f49aca66f96cf8e10b9a8 (table test-table [id=24aafb61db0d4e1f9bec85840f59bd0c]) successfully deleted
I20260927 18:37:14.004074 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19738
I20260927 18:37:14.022310 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19872
Sep 27 18:37:14 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260927 18:37:14.040187 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:14.023093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.97.66:40031)
0927 18:37:14.023273 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:14.023277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:14.023325 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:37:14.023541 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:14.023545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:14.023616 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:14.023699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:14.025508 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.026276 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.028221 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.028578 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.028663 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:14.034202 (+ 5539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:14.034216 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:14.034250 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:14.034282 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:14.037336 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:14.038249 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:14.038256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:14.038258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:14.038405 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:14.038825 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:14.038828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:14.038830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:14.039169 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:14.039379 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:14.039745 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 18:37:14.039910 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:37:14.040818 19995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:14.151728 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19872
I20260927 18:37:14.166627 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20005
Sep 27 18:37:14 dist-test-slave-n3p4 krb5kdc[19632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260927 18:37:14.185196 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:14.167633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.97.67:43785)
0927 18:37:14.167867 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:14.167886 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:14.167913 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:37:14.167998 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:14.168004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:14.168111 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:14.168240 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:14.169475 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.170779 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.172104 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.172908 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.173070 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:14.177542 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:14.177558 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:14.177576 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:14.177649 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:14.181949 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:14.182646 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:14.182659 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:14.182663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:14.182975 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:14.183340 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:14.183346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:14.183349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:14.183885 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:14.184209 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:14.184423 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:37:14.184683 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:37:14.186260 20128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:14.306598 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20005
I20260927 18:37:14.325644 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 19645
I20260927 18:37:14.326756 19707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:14.436507 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 19645
2026-09-27T18:37:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4234 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:14Z Disabled control of system clock
I20260927 18:37:14.512351 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36057
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:14.848753 20186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:14.849047 20186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:14.849196 20186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:14.856408 20186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:14.856516 20186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:14.856588 20186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:14.856679 20186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:14.856739 20186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:14.865876 20186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:14.867223 20186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:14.868975 20186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:14.880774 20194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:14.880853 20191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:14.880730 20192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:14.881698 20186 server_base.cc:1034] running on GCE node
I20260927 18:37:14.882526 20186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:14.883953 20186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:14.885190 20186 hybrid_clock.cc:648] HybridClock initialized: now 1790534234885129 us; error 58 us; skew 500 ppm
I20260927 18:37:14.885546 20186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:14.887961 20186 webserver.cc:460] Webserver started at http://127.0.97.126:37439/ using document root <none> and password file <none>
I20260927 18:37:14.888639 20186 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:14.888772 20186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:14.889086 20186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:14.892069 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "457525204cc24002aba03a2ea9c45557"
format_stamp: "Formatted at 2026-09-27 18:37:14 on dist-test-slave-n3p4"
server_key: "4c625d3b1db462fa8577047d2fc117c3"
server_key_iv: "5f2e9d3b4a5168bb22d16c00f00b7b3d"
server_key_version: "encryptionkey@0"
I20260927 18:37:14.892884 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "457525204cc24002aba03a2ea9c45557"
format_stamp: "Formatted at 2026-09-27 18:37:14 on dist-test-slave-n3p4"
server_key: "4c625d3b1db462fa8577047d2fc117c3"
server_key_iv: "5f2e9d3b4a5168bb22d16c00f00b7b3d"
server_key_version: "encryptionkey@0"
I20260927 18:37:14.898092 20186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:37:14.901547 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:14.902980 20186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:14.903173 20186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "457525204cc24002aba03a2ea9c45557"
format_stamp: "Formatted at 2026-09-27 18:37:14 on dist-test-slave-n3p4"
server_key: "4c625d3b1db462fa8577047d2fc117c3"
server_key_iv: "5f2e9d3b4a5168bb22d16c00f00b7b3d"
server_key_version: "encryptionkey@0"
I20260927 18:37:14.903409 20186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:14.931005 20186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:14.931926 20186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:14.932258 20186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:14.951040 20186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36057
I20260927 18:37:14.951061 20251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36057 every 8 connection(s)
I20260927 18:37:14.952409 20186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:14.955415 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20186
I20260927 18:37:14.955595 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:14.956123 389 external_mini_cluster.cc:1428] Setting key 66487711379e48d0af5d2e5705eb3de9
I20260927 18:37:14.956593 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:14.971401 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:14.958114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.97.126:36057)
0927 18:37:14.959138 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:14.959157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:14.959211 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:37:14.961476 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:14.961508 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:14.962118 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:14.962546 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:14.963114 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.965080 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.966436 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:14.967458 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:14.967638 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:14.968659 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:14.968669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:14.968685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:14.969425 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:14.969525 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:14.969550 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:14.969656 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:14.969846 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:37:14.970151 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260927 18:37:14.971699 20252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: Bootstrap starting.
I20260927 18:37:14.974795 20252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:14.975759 20252 log.cc:824] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:14.978365 20252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: No bootstrap required, opened a new log
I20260927 18:37:14.985947 20252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } }
I20260927 18:37:14.986327 20252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:14.986438 20252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457525204cc24002aba03a2ea9c45557, State: Initialized, Role: FOLLOWER
I20260927 18:37:14.987058 20252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } }
I20260927 18:37:14.987306 20252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:14.987447 20252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:14.987628 20252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:14.989725 20252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } }
I20260927 18:37:14.990272 20252 leader_election.cc:302] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 457525204cc24002aba03a2ea9c45557; no voters:
I20260927 18:37:14.990839 20252 leader_election.cc:288] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:14.991003 20257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:14.992354 20257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [term 1 LEADER]: Becoming Leader. State: Replica: 457525204cc24002aba03a2ea9c45557, State: Running, Role: LEADER
I20260927 18:37:14.992830 20257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } }
I20260927 18:37:14.992986 20252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:14.996418 20259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457525204cc24002aba03a2ea9c45557. Latest consensus state: current_term: 1 leader_uuid: "457525204cc24002aba03a2ea9c45557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } } }
I20260927 18:37:14.996790 20259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:14.997606 20258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "457525204cc24002aba03a2ea9c45557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457525204cc24002aba03a2ea9c45557" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36057 } } }
I20260927 18:37:14.997939 20258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:15.002516 20266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:15.007913 20266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:15.017459 20266 catalog_manager.cc:1347] Generated new cluster ID: df403529f1bf4b40b750452889b12322
I20260927 18:37:15.017599 20266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:15.048574 20266 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:15.050001 20266 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:15.060243 20266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: Generated new TSK 0
I20260927 18:37:15.061182 20266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:15.072535 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--builtin_ntp_servers=127.0.97.84:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:15.404752 20276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:15.405037 20276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:15.405124 20276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:15.412627 20276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:15.412730 20276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:15.412842 20276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260927 18:37:15.421988 20276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:15.423378 20276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:15.425022 20276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:15.437624 20284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:15.437706 20281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:15.437706 20282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:15.438863 20276 server_base.cc:1034] running on GCE node
I20260927 18:37:15.439465 20276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:15.440364 20276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:15.441613 20276 hybrid_clock.cc:648] HybridClock initialized: now 1790534235441559 us; error 64 us; skew 500 ppm
I20260927 18:37:15.442004 20276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:15.444365 20276 webserver.cc:460] Webserver started at http://127.0.97.65:45965/ using document root <none> and password file <none>
I20260927 18:37:15.445047 20276 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:15.445182 20276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:15.445518 20276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:15.448487 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data/instance:
uuid: "0e99df0a843b4e58bf16902a1120c139"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "9291d50994a29fe1b261d4271f991b67"
server_key_iv: "1c42421885195f795ea91c6b0a046957"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.449261 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0e99df0a843b4e58bf16902a1120c139"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "9291d50994a29fe1b261d4271f991b67"
server_key_iv: "1c42421885195f795ea91c6b0a046957"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.454221 20276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:37:15.457417 20290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.458700 20276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:15.458873 20276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal
uuid: "0e99df0a843b4e58bf16902a1120c139"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "9291d50994a29fe1b261d4271f991b67"
server_key_iv: "1c42421885195f795ea91c6b0a046957"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.459082 20276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:15.475036 20276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:15.475885 20276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:15.476203 20276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:15.477211 20276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:15.478749 20276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:15.478878 20276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.479000 20276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:15.479086 20276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.512392 20276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40849
I20260927 18:37:15.512445 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40849 every 8 connection(s)
I20260927 18:37:15.513710 20276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:15.516574 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20276
I20260927 18:37:15.516762 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:15.517285 389 external_mini_cluster.cc:1428] Setting key b8bbff23be88b5cb984bfe0d35b3314d
I20260927 18:37:15.520167 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--builtin_ntp_servers=127.0.97.84:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:15.528323 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:15.516230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47073 (local address 127.0.97.126:36057)
0927 18:37:15.516721 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:15.516727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:15.518191 (+ 1464us) server_negotiation.cc:407] Connection header received
0927 18:37:15.520555 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:15.520559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:15.520621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:15.520704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:15.522181 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.523051 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.525475 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.525924 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.526019 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:15.527608 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:15.527613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:15.527616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:15.527633 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:15.527669 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:15.527681 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:15.527752 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:15.527994 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 18:37:15.528068 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260927 18:37:15.529587 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:15.516871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36057 (local address 127.0.97.65:47073)
0927 18:37:15.518003 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:15.518065 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:15.520229 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:15.520978 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:15.520993 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:15.521124 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:15.521992 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:15.522019 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.523209 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.523217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:15.525328 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:15.525346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.526142 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.526151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:15.526510 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:15.527247 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:15.527253 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:15.527311 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:15.527414 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:15.527779 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:15.527792 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:15.527801 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:37:15.527982 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 18:37:15.528408 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":326,"threads_started":1}
I20260927 18:37:15.531165 20403 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36057
I20260927 18:37:15.531458 20403 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:15.532173 20403 heartbeater.cc:511] Master 127.0.97.126:36057 requested a full tablet report, sending...
I20260927 18:37:15.534200 20217 ts_manager.cc:194] Registered new tserver with Master: 0e99df0a843b4e58bf16902a1120c139 (127.0.97.65:40849)
I20260927 18:37:15.535686 20217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:47073
W20260927 18:37:15.846205 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:15.846524 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:15.846668 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:15.853734 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:15.853878 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:15.854058 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260927 18:37:15.863150 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:15.864439 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:15.866264 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:15.879333 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:15.879386 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:15.879333 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:15.879984 20407 server_base.cc:1034] running on GCE node
I20260927 18:37:15.880600 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:15.881549 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:15.882771 20407 hybrid_clock.cc:648] HybridClock initialized: now 1790534235882747 us; error 95 us; skew 500 ppm
I20260927 18:37:15.883106 20407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:15.885504 20407 webserver.cc:460] Webserver started at http://127.0.97.66:33169/ using document root <none> and password file <none>
I20260927 18:37:15.886247 20407 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:15.886386 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:15.886739 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:15.889622 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data/instance:
uuid: "67a361740f0540689d5a38e63b6795e2"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "278d00625509135da335291e9db455ef"
server_key_iv: "ee8e548ef5d759a95bd21ed9e8c593da"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.890476 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal/instance:
uuid: "67a361740f0540689d5a38e63b6795e2"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "278d00625509135da335291e9db455ef"
server_key_iv: "ee8e548ef5d759a95bd21ed9e8c593da"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.895473 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:37:15.899015 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.900314 20407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:15.900482 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal
uuid: "67a361740f0540689d5a38e63b6795e2"
format_stamp: "Formatted at 2026-09-27 18:37:15 on dist-test-slave-n3p4"
server_key: "278d00625509135da335291e9db455ef"
server_key_iv: "ee8e548ef5d759a95bd21ed9e8c593da"
server_key_version: "encryptionkey@0"
I20260927 18:37:15.900696 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:15.917264 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:15.918159 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:15.918440 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:15.919356 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:15.920819 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:15.920940 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.921073 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:15.921161 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:15.954777 20407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:40553
I20260927 18:37:15.954828 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:40553 every 8 connection(s)
I20260927 18:37:15.956010 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:15.956537 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20407
I20260927 18:37:15.956683 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:15.957185 389 external_mini_cluster.cc:1428] Setting key 0da72a487f233977891f0334b79e7fc5
I20260927 18:37:15.960187 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--builtin_ntp_servers=127.0.97.84:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:15.971448 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:15.959328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34857 (local address 127.0.97.126:36057)
0927 18:37:15.959548 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:15.959565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:15.961202 (+ 1637us) server_negotiation.cc:407] Connection header received
0927 18:37:15.963619 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:15.963628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:15.963718 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:15.963847 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:15.965426 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.966556 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.968614 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.969023 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.969152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:15.970580 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:15.970584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:15.970588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:15.970604 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:15.970633 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:15.970646 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:15.970759 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:15.971032 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:37:15.971134 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:37:15.972477 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:15.959956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36057 (local address 127.0.97.66:34857)
0927 18:37:15.961008 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:15.961055 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:15.963327 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:15.964263 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:15.964276 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:15.964407 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:15.965227 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:15.965255 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.966723 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.966731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:15.968462 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:15.968475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:15.969247 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:15.969255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:15.969487 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:15.970264 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:15.970267 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:15.970312 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:15.970386 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:15.970821 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:15.970830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:15.970836 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:37:15.971018 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 18:37:15.971383 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":341,"threads_started":1}
I20260927 18:37:15.973932 20534 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36057
I20260927 18:37:15.974251 20534 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:15.975018 20534 heartbeater.cc:511] Master 127.0.97.126:36057 requested a full tablet report, sending...
I20260927 18:37:15.976367 20217 ts_manager.cc:194] Registered new tserver with Master: 67a361740f0540689d5a38e63b6795e2 (127.0.97.66:40553)
I20260927 18:37:15.977133 20217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:34857
W20260927 18:37:16.293033 20537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:16.293349 20537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:16.293503 20537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:16.300992 20537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:16.301090 20537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:16.301280 20537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260927 18:37:16.311113 20537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:16.312362 20537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:16.314124 20537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:16.327491 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:16.327440 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:16.327467 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:16.331365 20537 server_base.cc:1034] running on GCE node
I20260927 18:37:16.331992 20537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:16.332897 20537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:16.334100 20537 hybrid_clock.cc:648] HybridClock initialized: now 1790534236334058 us; error 29 us; skew 500 ppm
I20260927 18:37:16.334435 20537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:37:16.336843 20537 webserver.cc:460] Webserver started at http://127.0.97.67:41953/ using document root <none> and password file <none>
I20260927 18:37:16.337538 20537 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:16.337656 20537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:16.338040 20537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:16.340879 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data/instance:
uuid: "93527c7c7c6640ab9f2ce2d9580ec88e"
format_stamp: "Formatted at 2026-09-27 18:37:16 on dist-test-slave-n3p4"
server_key: "4b9d2434ba06c0ae162a268561f242ac"
server_key_iv: "dc7c7423842d89f7c22a54c3dba4a520"
server_key_version: "encryptionkey@0"
I20260927 18:37:16.341645 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal/instance:
uuid: "93527c7c7c6640ab9f2ce2d9580ec88e"
format_stamp: "Formatted at 2026-09-27 18:37:16 on dist-test-slave-n3p4"
server_key: "4b9d2434ba06c0ae162a268561f242ac"
server_key_iv: "dc7c7423842d89f7c22a54c3dba4a520"
server_key_version: "encryptionkey@0"
I20260927 18:37:16.346710 20537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:37:16.350296 20551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:16.351749 20537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:37:16.352029 20537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal
uuid: "93527c7c7c6640ab9f2ce2d9580ec88e"
format_stamp: "Formatted at 2026-09-27 18:37:16 on dist-test-slave-n3p4"
server_key: "4b9d2434ba06c0ae162a268561f242ac"
server_key_iv: "dc7c7423842d89f7c22a54c3dba4a520"
server_key_version: "encryptionkey@0"
I20260927 18:37:16.352262 20537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:16.383589 20537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:16.384433 20537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:16.384712 20537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:16.385640 20537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:16.387184 20537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:16.387319 20537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:16.387455 20537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:16.387544 20537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:16.424559 20537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44011
I20260927 18:37:16.424647 20663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44011 every 8 connection(s)
I20260927 18:37:16.425899 20537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:16.429346 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20537
I20260927 18:37:16.429549 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:16.430163 389 external_mini_cluster.cc:1428] Setting key 61b70e1e902cea843c000caf4bd86886
I20260927 18:37:16.439105 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.428538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44777 (local address 127.0.97.126:36057)
0927 18:37:16.428716 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.428720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.430411 (+ 1691us) server_negotiation.cc:407] Connection header received
0927 18:37:16.432882 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.432887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.432978 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.433062 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:16.434243 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.435014 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.436506 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.436928 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.437022 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.438381 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:16.438386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:16.438389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:16.438408 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:16.438440 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:16.438452 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:16.438533 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.438784 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.438867 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:37:16.440011 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.429097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36057 (local address 127.0.97.67:44777)
0927 18:37:16.430201 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.430275 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.432563 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.433303 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.433318 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.433412 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:16.434078 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.434096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.435150 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.435154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.436380 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.436391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.437106 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.437110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.437301 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.438057 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:16.438060 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:16.438097 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:16.438174 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:16.438546 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:16.438553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:16.438558 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.438687 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.438962 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260927 18:37:16.441417 20664 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36057
I20260927 18:37:16.441723 20664 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:16.442765 20664 heartbeater.cc:511] Master 127.0.97.126:36057 requested a full tablet report, sending...
I20260927 18:37:16.444334 20217 ts_manager.cc:194] Registered new tserver with Master: 93527c7c7c6640ab9f2ce2d9580ec88e (127.0.97.67:44011)
I20260927 18:37:16.445226 20217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:44777
I20260927 18:37:16.445701 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:16.455597 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.449914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.0.97.126:36057)
0927 18:37:16.450102 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.450106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.450549 (+ 443us) server_negotiation.cc:407] Connection header received
0927 18:37:16.450658 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.450662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.450771 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.450856 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:16.451360 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.452121 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.453413 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.453918 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.454015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.454993 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:16.454997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:16.455000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:16.455017 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:16.455048 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:16.455061 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:16.455139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.455290 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.455361 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:37:16.459340 20217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:16.466568 20217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:16.501652 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.478838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40553 (local address 127.0.0.1:40558)
0927 18:37:16.493200 (+ 14362us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.493221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.493373 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.494274 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.494281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.494309 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:16.494794 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.494818 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.496707 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.496714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.498723 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.498737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.500630 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.500636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.500898 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.500958 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.501094 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.501273 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14183,"mutex_wait_us":23,"thread_start_us":14076,"threads_started":1}
I20260927 18:37:16.501802 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.479119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.97.66:40553)
0927 18:37:16.492089 (+ 12970us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.492102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.493516 (+ 1414us) server_negotiation.cc:407] Connection header received
0927 18:37:16.493638 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.493647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.494029 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.494275 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:16.495012 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.496543 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.498898 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.500403 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.500558 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.500836 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.501181 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.501370 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12720,"thread_start_us":12988,"threads_started":1}
I20260927 18:37:16.502838 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.478396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44011 (local address 127.0.0.1:49988)
0927 18:37:16.479579 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.479626 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.479833 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.495920 (+ 16087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.495931 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.495957 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:16.496527 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.496539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.498557 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.498562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.500330 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.500342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.502174 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.502180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.502393 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.502431 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.502493 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.502586 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":402,"thread_start_us":668,"threads_started":1}
I20260927 18:37:16.504025 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.478683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.0.97.67:44011)
0927 18:37:16.495359 (+ 16676us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.495365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.495396 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:37:16.495475 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.495480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.495650 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.496142 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:16.496736 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.498379 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.500527 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.501961 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.503092 (+ 1131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.503400 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.503468 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.503632 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16493,"thread_start_us":16309,"threads_started":1}
I20260927 18:37:16.506166 20469 tablet_service.cc:1467] Processing CreateTablet for tablet 5af997b6f2cc43b9827d66de07a753bf (DEFAULT_TABLE table=test-table [id=de89e505c1c44f5f944519c19d8538e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:16.507408 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af997b6f2cc43b9827d66de07a753bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:16.507544 20599 tablet_service.cc:1467] Processing CreateTablet for tablet 5af997b6f2cc43b9827d66de07a753bf (DEFAULT_TABLE table=test-table [id=de89e505c1c44f5f944519c19d8538e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:16.508176 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.480132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40849 (local address 127.0.0.1:53080)
0927 18:37:16.494911 (+ 14779us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.494928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.495029 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.495773 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.495780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.495814 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:16.496338 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.496356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.498554 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.498560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.500963 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.500976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.502949 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.502959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.507588 (+ 4629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.507624 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.507720 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.507857 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14644,"mutex_wait_us":41,"thread_start_us":14433,"threads_started":1}
I20260927 18:37:16.508704 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.480540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.0.97.65:40849)
0927 18:37:16.492866 (+ 12326us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.492876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.495168 (+ 2292us) server_negotiation.cc:407] Connection header received
0927 18:37:16.495282 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.495289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.495546 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.495733 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:16.496594 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.498402 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.501227 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.502733 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.502878 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.503367 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.507796 (+ 4429us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.507971 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12090,"thread_start_us":11952,"threads_started":1}
I20260927 18:37:16.509090 20599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af997b6f2cc43b9827d66de07a753bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:16.512048 20338 tablet_service.cc:1467] Processing CreateTablet for tablet 5af997b6f2cc43b9827d66de07a753bf (DEFAULT_TABLE table=test-table [id=de89e505c1c44f5f944519c19d8538e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:16.513316 20338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af997b6f2cc43b9827d66de07a753bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:16.524587 20682 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Bootstrap starting.
I20260927 18:37:16.526855 20683 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Bootstrap starting.
I20260927 18:37:16.528380 20684 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Bootstrap starting.
I20260927 18:37:16.528597 20682 tablet_bootstrap.cc:654] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:16.529767 20682 log.cc:824] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:16.531009 20683 tablet_bootstrap.cc:654] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:16.532191 20684 tablet_bootstrap.cc:654] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:16.532382 20683 log.cc:824] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:16.533449 20684 log.cc:824] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:16.535881 20683 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: No bootstrap required, opened a new log
I20260927 18:37:16.536262 20683 ts_tablet_manager.cc:1397] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 18:37:16.536842 20684 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: No bootstrap required, opened a new log
I20260927 18:37:16.537117 20684 ts_tablet_manager.cc:1397] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 18:37:16.538410 20403 heartbeater.cc:503] Master 127.0.97.126:36057 was elected leader, sending a full tablet report...
I20260927 18:37:16.539567 20682 tablet_bootstrap.cc:492] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: No bootstrap required, opened a new log
I20260927 18:37:16.540068 20682 ts_tablet_manager.cc:1397] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260927 18:37:16.548264 20684 raft_consensus.cc:348] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.548264 20683 raft_consensus.cc:348] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.548785 20683 raft_consensus.cc:374] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:16.548787 20684 raft_consensus.cc:374] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:16.548909 20684 raft_consensus.cc:729] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e99df0a843b4e58bf16902a1120c139, State: Initialized, Role: FOLLOWER
I20260927 18:37:16.548957 20683 raft_consensus.cc:729] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a361740f0540689d5a38e63b6795e2, State: Initialized, Role: FOLLOWER
I20260927 18:37:16.548838 20682 raft_consensus.cc:348] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.549311 20682 raft_consensus.cc:374] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:16.549454 20682 raft_consensus.cc:729] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93527c7c7c6640ab9f2ce2d9580ec88e, State: Initialized, Role: FOLLOWER
I20260927 18:37:16.549647 20684 consensus_queue.cc:260] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.549650 20683 consensus_queue.cc:260] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.550071 20682 consensus_queue.cc:260] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.551908 20684 ts_tablet_manager.cc:1428] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260927 18:37:16.553128 20664 heartbeater.cc:503] Master 127.0.97.126:36057 was elected leader, sending a full tablet report...
I20260927 18:37:16.553668 20682 ts_tablet_manager.cc:1428] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260927 18:37:16.555648 20534 heartbeater.cc:503] Master 127.0.97.126:36057 was elected leader, sending a full tablet report...
I20260927 18:37:16.556170 20683 ts_tablet_manager.cc:1428] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Time spent starting tablet: real 0.020s user 0.009s sys 0.009s
W20260927 18:37:16.677667 20665 tablet.cc:2369] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:16.709604 20535 tablet.cc:2369] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:16.766250 20404 tablet.cc:2369] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:16.898447 20690 raft_consensus.cc:482] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:16.898727 20690 raft_consensus.cc:504] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.900399 20690 leader_election.cc:288] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e99df0a843b4e58bf16902a1120c139 (127.0.97.65:40849), 67a361740f0540689d5a38e63b6795e2 (127.0.97.66:40553)
I20260927 18:37:16.906798 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.901007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49025 (local address 127.0.97.66:40553)
0927 18:37:16.901252 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.901260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.901533 (+ 273us) server_negotiation.cc:407] Connection header received
0927 18:37:16.901726 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.901731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.901880 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.901981 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:16.902491 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.903166 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.904977 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.905842 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.905994 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.906086 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.906394 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.906528 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:37:16.906870 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.900957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45571 (local address 127.0.97.65:40849)
0927 18:37:16.901168 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.901178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.901220 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:37:16.901311 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.901317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.901417 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.901544 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:16.902101 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.902831 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.904812 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.905861 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.906025 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.906112 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.906476 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.906597 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:37:16.906976 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.900677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40849 (local address 127.0.97.67:45571)
0927 18:37:16.900901 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.900921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.901050 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.901615 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.901621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.901651 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:16.901920 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.901929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.902992 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.903001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.904607 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.904620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.906051 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.906057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.906280 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.906303 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.906418 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.906573 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":96}
I20260927 18:37:16.906984 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.900686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40553 (local address 127.0.97.67:49025)
0927 18:37:16.901416 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:16.901433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:16.901581 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:16.901997 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:16.902001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:16.902023 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:16.902339 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.902347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.903325 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.903329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.904801 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:16.904814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.906017 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.906024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:16.906199 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.906222 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:37:16.906354 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 18:37:16.906504 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":238,"thread_start_us":363,"threads_started":1}
I20260927 18:37:16.908123 20358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af997b6f2cc43b9827d66de07a753bf" candidate_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e99df0a843b4e58bf16902a1120c139" is_pre_election: true
I20260927 18:37:16.908294 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af997b6f2cc43b9827d66de07a753bf" candidate_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a361740f0540689d5a38e63b6795e2" is_pre_election: true
I20260927 18:37:16.908725 20358 raft_consensus.cc:2454] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93527c7c7c6640ab9f2ce2d9580ec88e in term 0.
I20260927 18:37:16.908845 20489 raft_consensus.cc:2454] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93527c7c7c6640ab9f2ce2d9580ec88e in term 0.
I20260927 18:37:16.909386 20554 leader_election.cc:302] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e99df0a843b4e58bf16902a1120c139, 93527c7c7c6640ab9f2ce2d9580ec88e; no voters:
I20260927 18:37:16.909865 20690 raft_consensus.cc:2781] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:16.910022 20690 raft_consensus.cc:482] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:16.910149 20690 raft_consensus.cc:3037] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:16.912269 20690 raft_consensus.cc:504] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.912891 20690 leader_election.cc:288] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [CANDIDATE]: Term 1 election: Requested vote from peers 0e99df0a843b4e58bf16902a1120c139 (127.0.97.65:40849), 67a361740f0540689d5a38e63b6795e2 (127.0.97.66:40553)
I20260927 18:37:16.913286 20358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af997b6f2cc43b9827d66de07a753bf" candidate_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e99df0a843b4e58bf16902a1120c139"
I20260927 18:37:16.913352 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af997b6f2cc43b9827d66de07a753bf" candidate_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a361740f0540689d5a38e63b6795e2"
I20260927 18:37:16.913470 20358 raft_consensus.cc:3037] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:16.913627 20489 raft_consensus.cc:3037] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:16.915783 20358 raft_consensus.cc:2454] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93527c7c7c6640ab9f2ce2d9580ec88e in term 1.
I20260927 18:37:16.916090 20489 raft_consensus.cc:2454] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93527c7c7c6640ab9f2ce2d9580ec88e in term 1.
I20260927 18:37:16.916172 20554 leader_election.cc:302] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e99df0a843b4e58bf16902a1120c139, 93527c7c7c6640ab9f2ce2d9580ec88e; no voters:
I20260927 18:37:16.916474 20690 raft_consensus.cc:2781] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:16.916971 20690 raft_consensus.cc:686] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 1 LEADER]: Becoming Leader. State: Replica: 93527c7c7c6640ab9f2ce2d9580ec88e, State: Running, Role: LEADER
I20260927 18:37:16.917449 20690 consensus_queue.cc:237] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } }
I20260927 18:37:16.922247 20217 catalog_manager.cc:5495] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93527c7c7c6640ab9f2ce2d9580ec88e (127.0.97.67). New cstate: current_term: 1 leader_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93527c7c7c6640ab9f2ce2d9580ec88e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260927 18:37:16.946620 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:16.941451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.97.67:44011)
0927 18:37:16.941635 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:16.941639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:16.941654 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:37:16.941751 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:16.941754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:16.941893 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:16.942011 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:16.942329 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.943110 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.944434 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:16.944834 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:16.944935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:16.945410 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:16.946033 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:16.946140 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:16.946326 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 18:37:16.946395 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:37:16.955209 20358 raft_consensus.cc:1261] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 1 FOLLOWER]: Refusing update from remote peer 93527c7c7c6640ab9f2ce2d9580ec88e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:16.955336 20489 raft_consensus.cc:1261] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 1 FOLLOWER]: Refusing update from remote peer 93527c7c7c6640ab9f2ce2d9580ec88e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:16.956344 20690 consensus_queue.cc:1035] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a361740f0540689d5a38e63b6795e2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:16.956655 20692 consensus_queue.cc:1035] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e99df0a843b4e58bf16902a1120c139" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:16.981065 20697 mvcc.cc:205] Tried to move back new op lower bound from 7334028234557247488 to 7334028234417713152. Current Snapshot: MvccSnapshot[applied={T|T < 7334028234557247488}]
I20260927 18:37:16.998230 20216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:16.998524 20216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:17.002379 20216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 457525204cc24002aba03a2ea9c45557: Sending DeleteTablet for 3 replicas of tablet 5af997b6f2cc43b9827d66de07a753bf
I20260927 18:37:17.003777 20599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af997b6f2cc43b9827d66de07a753bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:16 UTC) from {username='slave'} at 127.0.0.1:49988
I20260927 18:37:17.003765 20469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af997b6f2cc43b9827d66de07a753bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:16 UTC) from {username='slave'} at 127.0.0.1:40558
I20260927 18:37:17.003973 20338 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af997b6f2cc43b9827d66de07a753bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:16 UTC) from {username='slave'} at 127.0.0.1:53080
I20260927 18:37:17.004982 20707 tablet_replica.cc:330] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: stopping tablet replica
I20260927 18:37:17.005602 20707 raft_consensus.cc:2229] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:17.006126 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20276
I20260927 18:37:17.006349 20707 raft_consensus.cc:2258] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:17.008643 20707 ts_tablet_manager.cc:1905] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:17.018054 20707 ts_tablet_manager.cc:1918] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:17.018325 20707 log.cc:1196] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-0/wal/wals/5af997b6f2cc43b9827d66de07a753bf
I20260927 18:37:17.018987 20707 ts_tablet_manager.cc:1939] T 5af997b6f2cc43b9827d66de07a753bf P 0e99df0a843b4e58bf16902a1120c139: Deleting consensus metadata
I20260927 18:37:17.021965 20203 catalog_manager.cc:4841] TS 0e99df0a843b4e58bf16902a1120c139 (127.0.97.65:40849): tablet 5af997b6f2cc43b9827d66de07a753bf (table test-table [id=de89e505c1c44f5f944519c19d8538e5]) successfully deleted
I20260927 18:37:17.024830 20705 tablet_replica.cc:330] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: stopping tablet replica
I20260927 18:37:17.025359 20705 raft_consensus.cc:2229] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:17.026057 20705 raft_consensus.cc:2258] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:17.026929 20706 tablet_replica.cc:330] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: stopping tablet replica
I20260927 18:37:17.027370 20706 raft_consensus.cc:2229] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:17.028024 20706 raft_consensus.cc:2258] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:17.029748 20706 ts_tablet_manager.cc:1905] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:17.030021 20705 ts_tablet_manager.cc:1905] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:17.034214 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:17.007998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.97.65:40849)
0927 18:37:17.008220 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:17.008228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:17.018308 (+ 10080us) server_negotiation.cc:407] Connection header received
0927 18:37:17.018931 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:17.018940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:17.019072 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:17.019161 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:17.020038 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.021523 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.024264 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.024801 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.024967 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:17.032414 (+ 7447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:17.032425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:17.032442 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:17.033240 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:17.033309 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:17.033342 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:17.033433 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:17.033688 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:37:17.033852 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:37:17.035578 20398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:17.036252 20706 ts_tablet_manager.cc:1918] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:17.036556 20706 log.cc:1196] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-2/wal/wals/5af997b6f2cc43b9827d66de07a753bf
I20260927 18:37:17.037063 20706 ts_tablet_manager.cc:1939] T 5af997b6f2cc43b9827d66de07a753bf P 93527c7c7c6640ab9f2ce2d9580ec88e: Deleting consensus metadata
I20260927 18:37:17.038827 20203 catalog_manager.cc:4841] TS 93527c7c7c6640ab9f2ce2d9580ec88e (127.0.97.67:44011): tablet 5af997b6f2cc43b9827d66de07a753bf (table test-table [id=de89e505c1c44f5f944519c19d8538e5]) successfully deleted
I20260927 18:37:17.042903 20705 ts_tablet_manager.cc:1918] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:17.043077 20705 log.cc:1196] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Deleting WAL directory at /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790533919012461-389-0/minicluster-data/ts-1/wal/wals/5af997b6f2cc43b9827d66de07a753bf
I20260927 18:37:17.043642 20705 ts_tablet_manager.cc:1939] T 5af997b6f2cc43b9827d66de07a753bf P 67a361740f0540689d5a38e63b6795e2: Deleting consensus metadata
I20260927 18:37:17.045519 20202 catalog_manager.cc:4841] TS 67a361740f0540689d5a38e63b6795e2 (127.0.97.66:40553): tablet 5af997b6f2cc43b9827d66de07a753bf (table test-table [id=de89e505c1c44f5f944519c19d8538e5]) successfully deleted
I20260927 18:37:17.127676 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20276
I20260927 18:37:17.146399 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20407
I20260927 18:37:17.154064 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:17.147249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.0.97.66:40553)
0927 18:37:17.147667 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:17.147672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:17.147687 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:37:17.147737 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:17.147741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:17.147837 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:17.147916 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:17.148387 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.149107 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.151005 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.151393 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.151484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:17.152641 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:17.152650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:17.152664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:17.153248 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:17.153295 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:17.153314 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:17.153407 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:17.153560 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:37:17.153643 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260927 18:37:17.154699 20529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:17.243860 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20407
I20260927 18:37:17.263396 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20537
I20260927 18:37:17.270816 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:17.264406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.0.97.67:44011)
0927 18:37:17.264602 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:17.264608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:17.264634 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:37:17.264710 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:17.264717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:17.264814 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:17.264944 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:17.265352 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.266278 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.267565 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:17.267998 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:17.268133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:17.269270 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:17.269281 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:17.269295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:17.270027 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:17.270085 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:17.270107 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:17.270229 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:17.270387 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:37:17.270480 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:37:17.271591 20659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:17.397989 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20537
I20260927 18:37:17.417263 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20186
I20260927 18:37:17.418483 20247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:17.529244 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20186
2026-09-27T18:37:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3093 ms)
[----------] 6 tests from EncryptionPolicyTest (21805 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:17 dist-test-slave-n3p4 krb5kdc[20715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:17 dist-test-slave-n3p4 krb5kdc[20715](info): set up 2 sockets
Sep 27 18:37:17 dist-test-slave-n3p4 krb5kdc[20715](info): commencing operation
krb5kdc: starting...
W20260927 18:37:18.658186 389 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:18 dist-test-slave-n3p4 krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:18.913547 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:19.243628 20728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:19.243916 20728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:19.244102 20728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:19.251256 20728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:19.251405 20728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:19.251492 20728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:19.251601 20728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:19.251662 20728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:19.261427 20728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:19.262749 20728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:19.264051 20728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:19.276064 20733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:19.276126 20734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:19.276371 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:19.277027 20728 server_base.cc:1034] running on GCE node
I20260927 18:37:19.277909 20728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:19.279285 20728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:19.280495 20728 hybrid_clock.cc:648] HybridClock initialized: now 1790534239280432 us; error 49 us; skew 500 ppm
I20260927 18:37:19.280869 20728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:19 dist-test-slave-n3p4 krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:19.284144 20728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:19.285725 20728 webserver.cc:460] Webserver started at http://127.0.0.1:33631/ using document root <none> and password file <none>
I20260927 18:37:19.286573 20728 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:19.286751 20728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:19.287099 20728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:19.290380 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "e05077f780ce42fab80292334428a1d3"
format_stamp: "Formatted at 2026-09-27 18:37:19 on dist-test-slave-n3p4"
server_key: "3081373476ea4b4e4eac251913b2f8da"
server_key_iv: "ee2730a7bfdc210bb5793fe8c205c599"
server_key_version: "encryptionkey@0"
I20260927 18:37:19.291369 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "e05077f780ce42fab80292334428a1d3"
format_stamp: "Formatted at 2026-09-27 18:37:19 on dist-test-slave-n3p4"
server_key: "3081373476ea4b4e4eac251913b2f8da"
server_key_iv: "ee2730a7bfdc210bb5793fe8c205c599"
server_key_version: "encryptionkey@0"
I20260927 18:37:19.297145 20728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:37:19.300729 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:19.302172 20728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:19.302376 20728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "e05077f780ce42fab80292334428a1d3"
format_stamp: "Formatted at 2026-09-27 18:37:19 on dist-test-slave-n3p4"
server_key: "3081373476ea4b4e4eac251913b2f8da"
server_key_iv: "ee2730a7bfdc210bb5793fe8c205c599"
server_key_version: "encryptionkey@0"
I20260927 18:37:19.302579 20728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:19.320915 20728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:19.325533 20728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:19.325856 20728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:19.345194 20728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33511
I20260927 18:37:19.345211 20794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33511 every 8 connection(s)
I20260927 18:37:19.346801 20728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:19.347539 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20728
I20260927 18:37:19.347822 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:19.348551 389 external_mini_cluster.cc:1428] Setting key 1aab1d1e5cc0616464860f333998d2f0
I20260927 18:37:19.351341 20795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:19.366370 20795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3: Bootstrap starting.
Sep 27 18:37:19 dist-test-slave-n3p4 krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:37:19.370671 20795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:19.372035 20795 log.cc:824] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:19.375936 20795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3: No bootstrap required, opened a new log
I20260927 18:37:19.377736 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:19.350606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40227 (local address 127.0.0.1:33511)
0927 18:37:19.351439 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:19.351454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:19.351502 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:37:19.353817 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:19.353852 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:19.354435 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:19.354942 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:19.357326 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:19.359432 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:19.361271 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:19.362380 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:19.362543 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:19.368233 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:19.368248 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:19.368270 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:19.368330 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:19.372617 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:19.373644 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:19.373654 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:19.373664 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:19.373884 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:19.374482 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:19.374492 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:19.374495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:19.375405 (+ 910us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:19.375714 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:19.376398 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 18:37:19.376738 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":346,"threads_started":1}
I20260927 18:37:19.382040 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:19.384806 20795 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260927 18:37:19.385185 20795 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:19.385334 20795 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05077f780ce42fab80292334428a1d3, State: Initialized, Role: FOLLOWER
I20260927 18:37:19.386026 20795 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260927 18:37:19.386298 20795 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:19.386505 20795 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:19.386760 20795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:19.389292 20795 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260927 18:37:19.389927 20795 leader_election.cc:302] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05077f780ce42fab80292334428a1d3; no voters:
I20260927 18:37:19.390612 20795 leader_election.cc:288] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:19.390668 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:19.392491 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [term 1 LEADER]: Becoming Leader. State: Replica: e05077f780ce42fab80292334428a1d3, State: Running, Role: LEADER
I20260927 18:37:19.392938 20795 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:19.393108 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260927 18:37:19.395855 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05077f780ce42fab80292334428a1d3. Latest consensus state: current_term: 1 leader_uuid: "e05077f780ce42fab80292334428a1d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } } }
I20260927 18:37:19.396255 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:19.396821 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05077f780ce42fab80292334428a1d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05077f780ce42fab80292334428a1d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } } }
I20260927 18:37:19.397107 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:19.397492 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:19.404651 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:19.406050 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:19.386819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41973 (local address 127.0.0.1:33511)
0927 18:37:19.386983 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:19.386988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:19.387509 (+ 521us) server_negotiation.cc:407] Connection header received
0927 18:37:19.387718 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:19.387723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:19.387806 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:19.387923 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:19.389351 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:19.390393 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:19.392287 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:19.393035 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:19.393159 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:19.397476 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:19.397485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:19.397492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:19.397539 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:19.401672 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:19.402719 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:19.402725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:19.402729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:19.402877 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:19.403434 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:19.403439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:19.403442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:19.403834 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:19.404122 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:19.405352 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0927 18:37:19.405525 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:19.420362 20813 catalog_manager.cc:1347] Generated new cluster ID: 95ca1e6a62f947799859fa707364735d
I20260927 18:37:19.420579 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:19.448217 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:19.449592 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:19.464965 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05077f780ce42fab80292334428a1d3: Generated new TSK 0
I20260927 18:37:19.465936 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:19.481860 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20728
I20260927 18:37:19.483764 20790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:19.580147 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20728
W20260927 18:37:19.595005 20713 connection.cc:535] client connection to 127.0.0.1:33511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33511): Connection reset by peer (error 104)
2026-09-27T18:37:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2047 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:19 dist-test-slave-n3p4 krb5kdc[20827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:19 dist-test-slave-n3p4 krb5kdc[20827](info): set up 2 sockets
Sep 27 18:37:19 dist-test-slave-n3p4 krb5kdc[20827](info): commencing operation
krb5kdc: starting...
W20260927 18:37:20.702167 389 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:20 dist-test-slave-n3p4 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:20.953516 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:21.282687 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:21.282974 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:21.283140 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:21.290158 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:21.290257 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:21.290345 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:21.290434 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:21.290522 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:21.299901 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:21.301219 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:21.302500 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:21.314950 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:21.314893 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:21.314981 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:21.315907 20840 server_base.cc:1034] running on GCE node
I20260927 18:37:21.316732 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:21.318163 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:21.319365 20840 hybrid_clock.cc:648] HybridClock initialized: now 1790534241319326 us; error 28 us; skew 500 ppm
I20260927 18:37:21.319763 20840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:21 dist-test-slave-n3p4 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:21.323046 20840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:21.324530 20840 webserver.cc:460] Webserver started at http://127.0.0.1:45141/ using document root <none> and password file <none>
I20260927 18:37:21.325205 20840 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:21.325343 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:21.325654 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:21.328552 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "373949f6565e44c0a71bf72453d94f9a"
format_stamp: "Formatted at 2026-09-27 18:37:21 on dist-test-slave-n3p4"
server_key: "8e10a01f4b7da111c67ed12d9a3ee3c8"
server_key_iv: "8643c384c01f1e8677b7c4123de4dece"
server_key_version: "encryptionkey@0"
I20260927 18:37:21.329299 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "373949f6565e44c0a71bf72453d94f9a"
format_stamp: "Formatted at 2026-09-27 18:37:21 on dist-test-slave-n3p4"
server_key: "8e10a01f4b7da111c67ed12d9a3ee3c8"
server_key_iv: "8643c384c01f1e8677b7c4123de4dece"
server_key_version: "encryptionkey@0"
I20260927 18:37:21.334338 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:37:21.337858 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.339246 20840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:21.339453 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "373949f6565e44c0a71bf72453d94f9a"
format_stamp: "Formatted at 2026-09-27 18:37:21 on dist-test-slave-n3p4"
server_key: "8e10a01f4b7da111c67ed12d9a3ee3c8"
server_key_iv: "8643c384c01f1e8677b7c4123de4dece"
server_key_version: "encryptionkey@0"
I20260927 18:37:21.339659 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:21.364421 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:21.369032 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:21.369325 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:21.388684 20840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44699
I20260927 18:37:21.388707 20906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44699 every 8 connection(s)
I20260927 18:37:21.390453 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:21.394171 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:21.397621 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20840
I20260927 18:37:21.397848 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:21.398379 389 external_mini_cluster.cc:1428] Setting key a43a8a3561578b3bec54fb07b014c9e2
I20260927 18:37:21.406637 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a: Bootstrap starting.
I20260927 18:37:21.410485 20907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:21.411689 20907 log.cc:824] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:21.411820 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.400162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59941 (local address 127.0.0.1:44699)
0927 18:37:21.400926 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.400936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.401014 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:37:21.402777 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.402799 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.403188 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.403502 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:21.404068 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.405384 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.407232 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.407888 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.408030 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.409429 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:21.409436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:21.409449 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:21.410107 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:21.410179 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:21.410202 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:21.410333 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.410676 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.410824 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":366,"threads_started":1}
I20260927 18:37:21.414764 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a: No bootstrap required, opened a new log
I20260927 18:37:21.415925 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:21.422667 20907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } }
I20260927 18:37:21.423113 20907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:21.423259 20907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373949f6565e44c0a71bf72453d94f9a, State: Initialized, Role: FOLLOWER
I20260927 18:37:21.423971 20907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } }
I20260927 18:37:21.424280 20907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:21.424419 20907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:21.424582 20907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:21.427696 20907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } }
I20260927 18:37:21.428011 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.420503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60973 (local address 127.0.0.1:44699)
0927 18:37:21.420785 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.420793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.421136 (+ 343us) server_negotiation.cc:407] Connection header received
0927 18:37:21.421399 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.421404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.421468 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.421602 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:21.422258 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.423038 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.425075 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.425655 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.425910 (+ 255us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.427181 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:21.427186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:21.427189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:21.427205 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:21.427240 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:21.427257 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:21.427344 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.427561 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.427603 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:37:21.428337 20907 leader_election.cc:302] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 373949f6565e44c0a71bf72453d94f9a; no voters:
I20260927 18:37:21.428997 20907 leader_election.cc:288] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:21.429129 20917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:21.430341 20917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [term 1 LEADER]: Becoming Leader. State: Replica: 373949f6565e44c0a71bf72453d94f9a, State: Running, Role: LEADER
I20260927 18:37:21.430867 20917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } }
I20260927 18:37:21.431355 20907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:21.433422 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "373949f6565e44c0a71bf72453d94f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } } }
I20260927 18:37:21.433732 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:21.433964 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 373949f6565e44c0a71bf72453d94f9a. Latest consensus state: current_term: 1 leader_uuid: "373949f6565e44c0a71bf72453d94f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373949f6565e44c0a71bf72453d94f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44699 } } }
I20260927 18:37:21.435094 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:21.436962 20924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:21.442178 20924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:21.452133 20924 catalog_manager.cc:1347] Generated new cluster ID: 7b6193097c6f456ebca451a51720fe05
I20260927 18:37:21.452265 20924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:21.465951 20924 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:21.467976 20924 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:21.478315 20924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 373949f6565e44c0a71bf72453d94f9a: Generated new TSK 0
I20260927 18:37:21.479349 20924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:21.490954 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20840
W20260927 18:37:21.491961 20902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59941
W20260927 18:37:21.492730 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:21.492939 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20840
W20260927 18:37:21.509894 20825 connection.cc:535] client connection to 127.0.0.1:44699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44699): Connection reset by peer (error 104)
2026-09-27T18:37:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:21 dist-test-slave-n3p4 krb5kdc[20938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:21 dist-test-slave-n3p4 krb5kdc[20938](info): set up 2 sockets
Sep 27 18:37:21 dist-test-slave-n3p4 krb5kdc[20938](info): commencing operation
krb5kdc: starting...
W20260927 18:37:22.602322 389 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:22 dist-test-slave-n3p4 krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:22.850265 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:23.172968 20951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:23.173224 20951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:23.173415 20951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:23.180531 20951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:23.180625 20951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:23.180737 20951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:23.180852 20951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:23.180913 20951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:23.190395 20951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:23.191599 20951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:23.192909 20951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:23.204571 20959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:23.204604 20957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:23.204535 20956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:23.205631 20951 server_base.cc:1034] running on GCE node
I20260927 18:37:23.206406 20951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:23.207758 20951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:23.209007 20951 hybrid_clock.cc:648] HybridClock initialized: now 1790534243208957 us; error 48 us; skew 500 ppm
I20260927 18:37:23.209355 20951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:23 dist-test-slave-n3p4 krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:23.215665 20951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:23.217164 20951 webserver.cc:460] Webserver started at http://127.0.0.1:42029/ using document root <none> and password file <none>
I20260927 18:37:23.217936 20951 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:23.218091 20951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:23.218402 20951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:23.221295 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "55fbf09145434b78b42f77cf98b20307"
format_stamp: "Formatted at 2026-09-27 18:37:23 on dist-test-slave-n3p4"
server_key: "a5f281d87cc1226dcab1e30ed4ed1ff2"
server_key_iv: "7755161de377ddd4188024451539f96d"
server_key_version: "encryptionkey@0"
I20260927 18:37:23.222131 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "55fbf09145434b78b42f77cf98b20307"
format_stamp: "Formatted at 2026-09-27 18:37:23 on dist-test-slave-n3p4"
server_key: "a5f281d87cc1226dcab1e30ed4ed1ff2"
server_key_iv: "7755161de377ddd4188024451539f96d"
server_key_version: "encryptionkey@0"
I20260927 18:37:23.227082 20951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:37:23.230474 20966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:23.231758 20951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:23.231943 20951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "55fbf09145434b78b42f77cf98b20307"
format_stamp: "Formatted at 2026-09-27 18:37:23 on dist-test-slave-n3p4"
server_key: "a5f281d87cc1226dcab1e30ed4ed1ff2"
server_key_iv: "7755161de377ddd4188024451539f96d"
server_key_version: "encryptionkey@0"
I20260927 18:37:23.232138 20951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:23.237722 20951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:23.241907 20951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:23.242188 20951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:23.261058 20951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36393
I20260927 18:37:23.261148 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36393 every 8 connection(s)
I20260927 18:37:23.262492 20951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:23.262915 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 20951
I20260927 18:37:23.263094 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:23.263677 389 external_mini_cluster.cc:1428] Setting key 8fd8abf256eb0847e09bc924fec735d8
I20260927 18:37:23.266899 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:23.271384 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:23.265477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47713 (local address 127.0.0.1:36393)
0927 18:37:23.266241 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:23.266268 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:23.266308 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:37:23.268020 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:23.268040 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:23.268362 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:23.268726 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:23.269882 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:23.269890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:23.269902 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:23.269915 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:23.269980 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:23.270000 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:23.270232 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:23.270376 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:37:23.270550 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":356,"threads_started":1}
I20260927 18:37:23.275061 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:23.278194 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307: Bootstrap starting.
I20260927 18:37:23.281594 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:23.281934 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:23.279300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46305 (local address 127.0.0.1:36393)
0927 18:37:23.279467 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:23.279475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:23.279847 (+ 372us) server_negotiation.cc:407] Connection header received
0927 18:37:23.280031 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:23.280036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:23.280088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:23.280204 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:23.281103 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:23.281108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:23.281113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:23.281134 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:23.281169 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:23.281184 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:23.281305 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:23.281466 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 18:37:23.281526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:37:23.282778 21018 log.cc:824] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:23.285349 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307: No bootstrap required, opened a new log
I20260927 18:37:23.293061 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } }
I20260927 18:37:23.293457 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:23.293567 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fbf09145434b78b42f77cf98b20307, State: Initialized, Role: FOLLOWER
I20260927 18:37:23.294207 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } }
I20260927 18:37:23.294445 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:23.294595 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:23.294778 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:23.297019 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } }
I20260927 18:37:23.297744 21018 leader_election.cc:302] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fbf09145434b78b42f77cf98b20307; no voters:
I20260927 18:37:23.298470 21018 leader_election.cc:288] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:23.298614 21028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:23.299371 21028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [term 1 LEADER]: Becoming Leader. State: Replica: 55fbf09145434b78b42f77cf98b20307, State: Running, Role: LEADER
I20260927 18:37:23.299896 21028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } }
I20260927 18:37:23.300660 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:23.303161 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fbf09145434b78b42f77cf98b20307. Latest consensus state: current_term: 1 leader_uuid: "55fbf09145434b78b42f77cf98b20307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } } }
I20260927 18:37:23.303489 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:23.304183 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fbf09145434b78b42f77cf98b20307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fbf09145434b78b42f77cf98b20307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36393 } } }
I20260927 18:37:23.304481 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:23.305045 21035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:23.309679 21035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:23.318143 21035 catalog_manager.cc:1347] Generated new cluster ID: 1616b5ec122649ef9c78dc54b9aaa4d1
I20260927 18:37:23.318264 21035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:23.344846 21035 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:23.345293 21035 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:23.351657 21035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fbf09145434b78b42f77cf98b20307: Generated new TSK 0
I20260927 18:37:23.352418 21035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:23.367558 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20951
W20260927 18:37:23.369550 21013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47713
W20260927 18:37:23.370383 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 20951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:23.370522 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 20951
2026-09-27T18:37:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:23 dist-test-slave-n3p4 krb5kdc[21049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:23 dist-test-slave-n3p4 krb5kdc[21049](info): set up 2 sockets
Sep 27 18:37:23 dist-test-slave-n3p4 krb5kdc[21049](info): commencing operation
krb5kdc: starting...
W20260927 18:37:24.482290 389 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:24 dist-test-slave-n3p4 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:24.746378 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:25.078191 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:25.078507 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:25.078722 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:25.086361 21062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:25.086540 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:25.086660 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:25.086782 21062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:25.086871 21062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:25.097265 21062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:25.098694 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:25.100087 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:25.112437 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:25.112524 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:25.112594 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:25.113178 21062 server_base.cc:1034] running on GCE node
I20260927 18:37:25.114084 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:25.115522 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:25.116797 21062 hybrid_clock.cc:648] HybridClock initialized: now 1790534245116753 us; error 52 us; skew 500 ppm
I20260927 18:37:25.117190 21062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:25 dist-test-slave-n3p4 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:25.120585 21062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:25.122186 21062 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
I20260927 18:37:25.122908 21062 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:25.123063 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:25.123431 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:25.126598 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "96cc69086d8a44868f4f1869992b0920"
format_stamp: "Formatted at 2026-09-27 18:37:25 on dist-test-slave-n3p4"
server_key: "6a7f29ec72801a0c6f8e070d2ec92971"
server_key_iv: "6dda59458bdd3774d3858cc8b010d3d1"
server_key_version: "encryptionkey@0"
I20260927 18:37:25.127421 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "96cc69086d8a44868f4f1869992b0920"
format_stamp: "Formatted at 2026-09-27 18:37:25 on dist-test-slave-n3p4"
server_key: "6a7f29ec72801a0c6f8e070d2ec92971"
server_key_iv: "6dda59458bdd3774d3858cc8b010d3d1"
server_key_version: "encryptionkey@0"
I20260927 18:37:25.133193 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:37:25.136863 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:25.138202 21062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:25.138405 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "96cc69086d8a44868f4f1869992b0920"
format_stamp: "Formatted at 2026-09-27 18:37:25 on dist-test-slave-n3p4"
server_key: "6a7f29ec72801a0c6f8e070d2ec92971"
server_key_iv: "6dda59458bdd3774d3858cc8b010d3d1"
server_key_version: "encryptionkey@0"
I20260927 18:37:25.138614 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:25.155047 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:25.159294 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:25.159608 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:25.178725 21062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37647
I20260927 18:37:25.178817 21128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37647 every 8 connection(s)
I20260927 18:37:25.180074 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:25.181449 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21062
I20260927 18:37:25.181641 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:25.182197 389 external_mini_cluster.cc:1428] Setting key 405503c658aa302645a42d2704e3035b
I20260927 18:37:25.184751 21129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:37:25 dist-test-slave-n3p4 krb5kdc[21049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:37:25.196194 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920: Bootstrap starting.
I20260927 18:37:25.199736 21129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:25.200855 21129 log.cc:824] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:25.204083 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920: No bootstrap required, opened a new log
I20260927 18:37:25.204545 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:25.184005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37251 (local address 127.0.0.1:37647)
0927 18:37:25.184765 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:25.184776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:25.184852 (+ 76us) server_negotiation.cc:407] Connection header received
0927 18:37:25.186731 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:25.186788 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:25.187160 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:25.187600 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:25.189049 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:25.190550 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:25.191860 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:25.192524 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:25.192739 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:25.197159 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:25.197167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:25.197186 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:25.197226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:25.200442 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:25.201206 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:25.201212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:25.201218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:25.201381 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:25.201813 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:25.201820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:25.201824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:25.202341 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:25.202648 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:25.203118 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 18:37:25.203531 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":360,"threads_started":1}
I20260927 18:37:25.208241 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:25.213280 21129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 18:37:25.213682 21129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:25.213842 21129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cc69086d8a44868f4f1869992b0920, State: Initialized, Role: FOLLOWER
I20260927 18:37:25.214432 21129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 18:37:25.214704 21129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:25.214874 21129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:25.215073 21129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:25.217527 21129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 18:37:25.218168 21129 leader_election.cc:302] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96cc69086d8a44868f4f1869992b0920; no voters:
I20260927 18:37:25.218823 21129 leader_election.cc:288] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:25.218945 21139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:25.220405 21139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [term 1 LEADER]: Becoming Leader. State: Replica: 96cc69086d8a44868f4f1869992b0920, State: Running, Role: LEADER
I20260927 18:37:25.220976 21139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 18:37:25.221318 21129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:25.223691 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96cc69086d8a44868f4f1869992b0920. Latest consensus state: current_term: 1 leader_uuid: "96cc69086d8a44868f4f1869992b0920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } } }
I20260927 18:37:25.224086 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:25.224604 21140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96cc69086d8a44868f4f1869992b0920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cc69086d8a44868f4f1869992b0920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } } }
I20260927 18:37:25.224902 21140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:25.231737 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:25.212595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35639 (local address 127.0.0.1:37647)
0927 18:37:25.212799 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:25.212807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:25.213171 (+ 364us) server_negotiation.cc:407] Connection header received
0927 18:37:25.213369 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:25.213375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:25.213439 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:25.213545 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:25.215394 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:25.216137 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:25.217359 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:25.217885 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:25.218020 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:25.221248 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:25.221254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:25.221259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:25.221305 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:25.226160 (+ 4855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:25.227403 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:25.227412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:25.227419 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:25.227671 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:25.228307 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:25.228316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:25.228322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:25.228839 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:25.229203 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:25.230711 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0927 18:37:25.231084 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:25.238947 21144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:25.246690 21144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:25.256892 21144 catalog_manager.cc:1347] Generated new cluster ID: f02fa15981f440089e00b7aad7ba03c2
I20260927 18:37:25.257068 21144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:25.274734 21144 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:25.276114 21144 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:25.288228 21144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96cc69086d8a44868f4f1869992b0920: Generated new TSK 0
I20260927 18:37:25.289393 21144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:25.302296 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21062
I20260927 18:37:25.303583 21124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:25.399796 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21062
2026-09-27T18:37:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:25 dist-test-slave-n3p4 krb5kdc[21162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:25 dist-test-slave-n3p4 krb5kdc[21162](info): set up 2 sockets
Sep 27 18:37:25 dist-test-slave-n3p4 krb5kdc[21162](info): commencing operation
krb5kdc: starting...
W20260927 18:37:26.502199 389 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:26 dist-test-slave-n3p4 krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:26.747076 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:27.068894 21175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:27.069155 21175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:27.069314 21175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:27.076337 21175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:27.076435 21175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:27.076540 21175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:27.076659 21175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:27.076733 21175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:27.086374 21175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:27.087679 21175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:27.088959 21175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:27.100117 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:27.100296 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:27.100327 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:27.101212 21175 server_base.cc:1034] running on GCE node
I20260927 18:37:27.102020 21175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:27.103428 21175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:27.104628 21175 hybrid_clock.cc:648] HybridClock initialized: now 1790534247104580 us; error 36 us; skew 500 ppm
I20260927 18:37:27.104992 21175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:27 dist-test-slave-n3p4 krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:27.108085 21175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:27.109572 21175 webserver.cc:460] Webserver started at http://127.0.0.1:41325/ using document root <none> and password file <none>
I20260927 18:37:27.110280 21175 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:27.110409 21175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:27.110707 21175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:27.113525 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "f3277ab2d7dc4759a978f55d4d4d454a"
format_stamp: "Formatted at 2026-09-27 18:37:27 on dist-test-slave-n3p4"
server_key: "61b37c034f0e20353ad587ccac20dc93"
server_key_iv: "bf12443d27d32cac64b85fc0c709e7fc"
server_key_version: "encryptionkey@0"
I20260927 18:37:27.114262 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "f3277ab2d7dc4759a978f55d4d4d454a"
format_stamp: "Formatted at 2026-09-27 18:37:27 on dist-test-slave-n3p4"
server_key: "61b37c034f0e20353ad587ccac20dc93"
server_key_iv: "bf12443d27d32cac64b85fc0c709e7fc"
server_key_version: "encryptionkey@0"
I20260927 18:37:27.119107 21175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:37:27.122344 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:27.123586 21175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:37:27.123752 21175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "f3277ab2d7dc4759a978f55d4d4d454a"
format_stamp: "Formatted at 2026-09-27 18:37:27 on dist-test-slave-n3p4"
server_key: "61b37c034f0e20353ad587ccac20dc93"
server_key_iv: "bf12443d27d32cac64b85fc0c709e7fc"
server_key_version: "encryptionkey@0"
I20260927 18:37:27.123950 21175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:27.152842 21175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:27.157195 21175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:27.157480 21175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:27.175510 21175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37863
I20260927 18:37:27.175568 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37863 every 8 connection(s)
I20260927 18:37:27.176842 21175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:27.179672 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21175
I20260927 18:37:27.179879 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:27.180388 389 external_mini_cluster.cc:1428] Setting key 4b99562965240a1f10ffade6860af6b9
I20260927 18:37:27.180876 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:27.194013 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:27.182160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50003 (local address 127.0.0.1:37863)
0927 18:37:27.182937 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:27.182948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:27.182989 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:37:27.185247 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:27.185267 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:27.185640 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:27.186042 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:27.186609 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:27.187927 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:27.189742 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:27.190411 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:27.190552 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:27.191730 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:27.191737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:27.191748 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:27.192328 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:27.192397 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:27.192419 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:27.192504 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:27.192715 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:37:27.192996 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":367,"threads_started":1}
I20260927 18:37:27.194401 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a: Bootstrap starting.
I20260927 18:37:27.197742 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:27.197887 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:27.199026 21242 log.cc:824] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:27.202196 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a: No bootstrap required, opened a new log
I20260927 18:37:27.209165 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:27.202702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58171 (local address 127.0.0.1:37863)
0927 18:37:27.202877 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:27.202884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:27.203376 (+ 492us) server_negotiation.cc:407] Connection header received
0927 18:37:27.203531 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:27.203537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:27.203611 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:27.203723 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:27.204145 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:27.205273 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:27.206592 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:27.207332 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:27.207480 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:27.208415 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:27.208422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:27.208427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:27.208446 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:27.208488 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:27.208512 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:27.208593 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:27.208715 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:37:27.208819 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:37:27.211508 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260927 18:37:27.211865 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:27.211972 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3277ab2d7dc4759a978f55d4d4d454a, State: Initialized, Role: FOLLOWER
I20260927 18:37:27.212523 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260927 18:37:27.212739 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:27.212852 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:27.212962 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:27.215452 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260927 18:37:27.215878 21242 leader_election.cc:302] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3277ab2d7dc4759a978f55d4d4d454a; no voters:
I20260927 18:37:27.216405 21242 leader_election.cc:288] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:27.216522 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:27.217224 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [term 1 LEADER]: Becoming Leader. State: Replica: f3277ab2d7dc4759a978f55d4d4d454a, State: Running, Role: LEADER
I20260927 18:37:27.217860 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260927 18:37:27.218631 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:27.220804 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260927 18:37:27.221477 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:27.221879 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3277ab2d7dc4759a978f55d4d4d454a. Latest consensus state: current_term: 1 leader_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3277ab2d7dc4759a978f55d4d4d454a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260927 18:37:27.226955 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:27.228257 21261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:27.233608 21261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:27.242489 21261 catalog_manager.cc:1347] Generated new cluster ID: 42cdd519a8f64d29b04041bfc6f88c31
I20260927 18:37:27.242616 21261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:27.261736 21261 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:27.263085 21261 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:27.271799 21261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3277ab2d7dc4759a978f55d4d4d454a: Generated new TSK 0
I20260927 18:37:27.272625 21261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:27.277228 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21175
W20260927 18:37:27.278908 21237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50003
W20260927 18:37:27.279531 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21175: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:27.279687 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21175
2026-09-27T18:37:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:27 dist-test-slave-n3p4 krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:27 dist-test-slave-n3p4 krb5kdc[21273](info): set up 2 sockets
Sep 27 18:37:27 dist-test-slave-n3p4 krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260927 18:37:28.374001 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:28 dist-test-slave-n3p4 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:28.621428 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:28.941763 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:28.942112 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:28.942276 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:28.949240 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:28.949378 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:28.949481 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:28.949585 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:28.949668 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:28.959131 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:28.960391 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:28.961648 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:28.974140 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:28.974179 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:28.974371 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:28.975207 21286 server_base.cc:1034] running on GCE node
I20260927 18:37:28.975982 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:28.977319 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:28.978536 21286 hybrid_clock.cc:648] HybridClock initialized: now 1790534248978490 us; error 42 us; skew 500 ppm
I20260927 18:37:28.978886 21286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:28 dist-test-slave-n3p4 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:28.986167 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:28.987596 21286 webserver.cc:460] Webserver started at http://127.0.0.1:37019/ using document root <none> and password file <none>
I20260927 18:37:28.988226 21286 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:28.988320 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:28.988562 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:28.991460 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1"
format_stamp: "Formatted at 2026-09-27 18:37:28 on dist-test-slave-n3p4"
server_key: "6d66fb06ef63936959c80ecc698dda82"
server_key_iv: "dc75404418539870b9650825b26733a6"
server_key_version: "encryptionkey@0"
I20260927 18:37:28.992152 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1"
format_stamp: "Formatted at 2026-09-27 18:37:28 on dist-test-slave-n3p4"
server_key: "6d66fb06ef63936959c80ecc698dda82"
server_key_iv: "dc75404418539870b9650825b26733a6"
server_key_version: "encryptionkey@0"
I20260927 18:37:28.997139 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 18:37:29.000427 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:29.001684 21286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:29.001888 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1"
format_stamp: "Formatted at 2026-09-27 18:37:28 on dist-test-slave-n3p4"
server_key: "6d66fb06ef63936959c80ecc698dda82"
server_key_iv: "dc75404418539870b9650825b26733a6"
server_key_version: "encryptionkey@0"
I20260927 18:37:29.002049 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:29.007565 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:29.011431 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:29.011659 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:29.029965 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46145
I20260927 18:37:29.029989 21352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46145 every 8 connection(s)
I20260927 18:37:29.031350 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:29.033231 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21286
I20260927 18:37:29.033427 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:29.034042 389 external_mini_cluster.cc:1428] Setting key 474cd12cc549b94373e224e643a7f0a8
I20260927 18:37:29.035669 21353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:29.041539 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:29.035811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57111 (local address 127.0.0.1:46145)
0927 18:37:29.036546 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:29.036558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:29.036597 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:37:29.038255 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:29.038276 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:29.038617 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:29.039026 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:29.040011 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:29.040017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:29.040034 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:29.040049 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:29.040113 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:29.040135 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:29.040361 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:29.040491 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:37:29.040665 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":306,"threads_started":1}
I20260927 18:37:29.044921 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:29.047286 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1: Bootstrap starting.
I20260927 18:37:29.050943 21353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:29.051416 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:29.049061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34977 (local address 127.0.0.1:46145)
0927 18:37:29.049244 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:29.049250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:29.049524 (+ 274us) server_negotiation.cc:407] Connection header received
0927 18:37:29.049689 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:29.049692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:29.049735 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:29.049880 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:29.050749 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:29.050753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:29.050757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:29.050769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:29.050804 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:29.050821 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:29.050914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:29.051074 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:37:29.051129 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:29.052156 21353 log.cc:824] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:29.054809 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1: No bootstrap required, opened a new log
I20260927 18:37:29.062018 21353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260927 18:37:29.062410 21353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:29.062520 21353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2349a76ec62044bcbe0aa0b4ad2e15c1, State: Initialized, Role: FOLLOWER
I20260927 18:37:29.063151 21353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260927 18:37:29.063391 21353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:29.063529 21353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:29.063694 21353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:29.065961 21353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260927 18:37:29.066551 21353 leader_election.cc:302] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2349a76ec62044bcbe0aa0b4ad2e15c1; no voters:
I20260927 18:37:29.067191 21353 leader_election.cc:288] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:29.067281 21363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:29.068017 21363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [term 1 LEADER]: Becoming Leader. State: Replica: 2349a76ec62044bcbe0aa0b4ad2e15c1, State: Running, Role: LEADER
I20260927 18:37:29.068605 21363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260927 18:37:29.069311 21353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:29.071534 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2349a76ec62044bcbe0aa0b4ad2e15c1. Latest consensus state: current_term: 1 leader_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } } }
I20260927 18:37:29.071811 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2349a76ec62044bcbe0aa0b4ad2e15c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } } }
I20260927 18:37:29.072158 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:29.072237 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:29.073643 21371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:29.078814 21371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:29.087472 21371 catalog_manager.cc:1347] Generated new cluster ID: ddc6673a258d42edbf453cd5227a75f3
I20260927 18:37:29.087579 21371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:29.105324 21371 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:29.105732 21371 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:29.117560 21371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2349a76ec62044bcbe0aa0b4ad2e15c1: Generated new TSK 0
I20260927 18:37:29.118345 21371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:29.131134 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21286
W20260927 18:37:29.132707 21348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57111
W20260927 18:37:29.133493 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:29.133646 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21286
2026-09-27T18:37:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:29 dist-test-slave-n3p4 krb5kdc[21384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:29 dist-test-slave-n3p4 krb5kdc[21384](info): set up 2 sockets
Sep 27 18:37:29 dist-test-slave-n3p4 krb5kdc[21384](info): commencing operation
krb5kdc: starting...
W20260927 18:37:30.246167 389 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:30 dist-test-slave-n3p4 krb5kdc[21384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:30.493675 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:30.820354 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:30.820688 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:30.820868 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:30.828217 21397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:30.828373 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:30.828502 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:30.828609 21397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:30.828692 21397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:30.839282 21397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:30.840579 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:30.841980 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:30.854969 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:30.855047 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:30.854998 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:30.856365 21397 server_base.cc:1034] running on GCE node
I20260927 18:37:30.857115 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:30.858566 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:30.859760 21397 hybrid_clock.cc:648] HybridClock initialized: now 1790534250859719 us; error 46 us; skew 500 ppm
I20260927 18:37:30.860122 21397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:30 dist-test-slave-n3p4 krb5kdc[21384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:30.863291 21397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:30.864763 21397 webserver.cc:460] Webserver started at http://127.0.0.1:34601/ using document root <none> and password file <none>
I20260927 18:37:30.865442 21397 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:30.865584 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:30.865996 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:30.868927 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "077136b47f3647a2ad8c3f6d4b073403"
format_stamp: "Formatted at 2026-09-27 18:37:30 on dist-test-slave-n3p4"
server_key: "c4d65adc77a318b068a2fbde8eddfbd3"
server_key_iv: "0811395baaf7dfd7bd99ea9a56b68cb3"
server_key_version: "encryptionkey@0"
I20260927 18:37:30.869671 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "077136b47f3647a2ad8c3f6d4b073403"
format_stamp: "Formatted at 2026-09-27 18:37:30 on dist-test-slave-n3p4"
server_key: "c4d65adc77a318b068a2fbde8eddfbd3"
server_key_iv: "0811395baaf7dfd7bd99ea9a56b68cb3"
server_key_version: "encryptionkey@0"
I20260927 18:37:30.874684 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:37:30.878333 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:30.879794 21397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:37:30.880002 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "077136b47f3647a2ad8c3f6d4b073403"
format_stamp: "Formatted at 2026-09-27 18:37:30 on dist-test-slave-n3p4"
server_key: "c4d65adc77a318b068a2fbde8eddfbd3"
server_key_iv: "0811395baaf7dfd7bd99ea9a56b68cb3"
server_key_version: "encryptionkey@0"
I20260927 18:37:30.880204 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:30.910916 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:30.915056 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:30.915343 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:30.933707 21397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43433
I20260927 18:37:30.933740 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43433 every 8 connection(s)
I20260927 18:37:30.935142 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:30.937259 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21397
I20260927 18:37:30.937464 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:30.937983 389 external_mini_cluster.cc:1428] Setting key eefc70f65d89329a4288d1f4a4f7d1f9
I20260927 18:37:30.939234 21464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:37:30 dist-test-slave-n3p4 krb5kdc[21384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:37:30.952524 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403: Bootstrap starting.
I20260927 18:37:30.956521 21464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:30.957710 21464 log.cc:824] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:30.961061 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403: No bootstrap required, opened a new log
I20260927 18:37:30.961645 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:30.939836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:54419 (local address 127.0.0.1:43433)
0927 18:37:30.940588 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:30.940603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:30.940654 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:37:30.942575 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:30.942602 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:30.943117 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:30.943542 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:30.944839 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:30.946920 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:30.948121 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:30.949128 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:30.949307 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:30.953275 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:30.953285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:30.953307 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:30.953352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:30.957167 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:30.957898 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:30.957908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:30.957915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:30.958105 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:30.958637 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:30.958641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:30.958643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:30.959215 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:30.959527 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:30.960069 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 18:37:30.960389 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":311,"threads_started":1}
I20260927 18:37:30.965380 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:30.969111 21464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 18:37:30.969463 21464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:30.969580 21464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077136b47f3647a2ad8c3f6d4b073403, State: Initialized, Role: FOLLOWER
I20260927 18:37:30.970230 21464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 18:37:30.970466 21464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:30.970620 21464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:30.970821 21464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:30.973397 21464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 18:37:30.973941 21464 leader_election.cc:302] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077136b47f3647a2ad8c3f6d4b073403; no voters:
I20260927 18:37:30.974592 21464 leader_election.cc:288] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:30.974689 21474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:30.975993 21474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [term 1 LEADER]: Becoming Leader. State: Replica: 077136b47f3647a2ad8c3f6d4b073403, State: Running, Role: LEADER
I20260927 18:37:30.976567 21474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 18:37:30.977025 21464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:30.983639 21476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077136b47f3647a2ad8c3f6d4b073403. Latest consensus state: current_term: 1 leader_uuid: "077136b47f3647a2ad8c3f6d4b073403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } } }
I20260927 18:37:30.984166 21476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:30.985258 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:30.969884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:35205 (local address 127.0.0.1:43433)
0927 18:37:30.970055 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:30.970062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:30.970558 (+ 496us) server_negotiation.cc:407] Connection header received
0927 18:37:30.970748 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:30.970753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:30.970849 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:30.970960 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:30.972205 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:30.973281 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:30.974661 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:30.975486 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:30.975596 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:30.978308 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:30.978319 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:30.978322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:30.978351 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:30.981116 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:30.981833 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:30.981837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:30.981840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:30.982028 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:30.982644 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:30.982649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:30.982652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:30.983032 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:30.984397 (+ 1365us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:30.984671 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 18:37:30.984846 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:37:30.985190 21475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077136b47f3647a2ad8c3f6d4b073403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077136b47f3647a2ad8c3f6d4b073403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } } }
I20260927 18:37:30.987380 21475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:30.988862 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:30.993876 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:31.002595 21480 catalog_manager.cc:1347] Generated new cluster ID: 3115286ea5e8442e96bbcc807f097512
I20260927 18:37:31.002708 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:31.033474 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:31.035307 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:31.044366 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077136b47f3647a2ad8c3f6d4b073403: Generated new TSK 0
I20260927 18:37:31.045349 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:31.062671 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21397
I20260927 18:37:31.063861 21459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:31.154155 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21397
W20260927 18:37:31.171231 21382 connection.cc:535] client connection to 127.0.0.1:43433 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43433): Connection reset by peer (error 104)
2026-09-27T18:37:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:31 dist-test-slave-n3p4 krb5kdc[21496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:31 dist-test-slave-n3p4 krb5kdc[21496](info): set up 2 sockets
Sep 27 18:37:31 dist-test-slave-n3p4 krb5kdc[21496](info): commencing operation
krb5kdc: starting...
W20260927 18:37:32.250168 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:32 dist-test-slave-n3p4 krb5kdc[21496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:32.493170 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:32.825806 21509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:32.826105 21509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:32.826215 21509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:32.833395 21509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:32.833503 21509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:32.833555 21509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:32.833616 21509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:32.833657 21509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:32.843464 21509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:32.844758 21509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:32.846083 21509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:32.859522 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:32.859570 21515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:32.859594 21517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:32.863238 21509 server_base.cc:1034] running on GCE node
I20260927 18:37:32.864122 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:32.865662 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:32.866881 21509 hybrid_clock.cc:648] HybridClock initialized: now 1790534252866823 us; error 54 us; skew 500 ppm
I20260927 18:37:32.867231 21509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:32 dist-test-slave-n3p4 krb5kdc[21496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:32.870204 21509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:32.871748 21509 webserver.cc:460] Webserver started at http://127.0.0.1:37833/ using document root <none> and password file <none>
I20260927 18:37:32.872447 21509 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:32.872587 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:32.872881 21509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:32.875891 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "8895f9361cd74b4e938b7fa47893ddc3"
format_stamp: "Formatted at 2026-09-27 18:37:32 on dist-test-slave-n3p4"
server_key: "95551a995f82f1b0ffc1566a12e6a636"
server_key_iv: "23b819efb3087951370f4cdbc480e8e2"
server_key_version: "encryptionkey@0"
I20260927 18:37:32.876658 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "8895f9361cd74b4e938b7fa47893ddc3"
format_stamp: "Formatted at 2026-09-27 18:37:32 on dist-test-slave-n3p4"
server_key: "95551a995f82f1b0ffc1566a12e6a636"
server_key_iv: "23b819efb3087951370f4cdbc480e8e2"
server_key_version: "encryptionkey@0"
I20260927 18:37:32.881623 21509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:37:32.885128 21524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:32.886556 21509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:32.886735 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "8895f9361cd74b4e938b7fa47893ddc3"
format_stamp: "Formatted at 2026-09-27 18:37:32 on dist-test-slave-n3p4"
server_key: "95551a995f82f1b0ffc1566a12e6a636"
server_key_iv: "23b819efb3087951370f4cdbc480e8e2"
server_key_version: "encryptionkey@0"
I20260927 18:37:32.886912 21509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:32.904493 21509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:32.908921 21509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:32.909199 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:32.927565 21509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45725
I20260927 18:37:32.927587 21575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45725 every 8 connection(s)
I20260927 18:37:32.928968 21509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:32.932611 21576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:32.936304 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21509
I20260927 18:37:32.936498 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:32.936939 389 external_mini_cluster.cc:1428] Setting key bf7f30b375a8db9ad5eb7c4038cc8c1c
I20260927 18:37:32.946163 21576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3: Bootstrap starting.
I20260927 18:37:32.950205 21576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:32.951406 21576 log.cc:824] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:32.951766 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:32.938851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:53429 (local address 127.0.0.1:45725)
0927 18:37:32.939607 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:32.939641 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:32.939693 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:37:32.941538 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:32.941568 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:32.942191 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:32.942630 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:32.943177 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:32.945160 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:32.946598 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:32.947580 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:32.947775 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:32.949143 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:32.949153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:32.949172 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:32.949914 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:32.950010 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:32.950052 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:32.950229 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:32.950528 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:37:32.950712 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":328,"threads_started":1}
I20260927 18:37:32.954334 21576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3: No bootstrap required, opened a new log
I20260927 18:37:32.955628 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:32.962697 21576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260927 18:37:32.963145 21576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:32.963265 21576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8895f9361cd74b4e938b7fa47893ddc3, State: Initialized, Role: FOLLOWER
I20260927 18:37:32.963887 21576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260927 18:37:32.964154 21576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:32.964352 21576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:32.964550 21576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:32.966856 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:32.959899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:57941 (local address 127.0.0.1:45725)
0927 18:37:32.960139 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:32.960152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:32.960557 (+ 405us) server_negotiation.cc:407] Connection header received
0927 18:37:32.960714 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:32.960717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:32.960818 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:32.960902 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:32.961507 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:32.962339 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:32.964202 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:32.964806 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:32.964920 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:32.966084 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:32.966088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:32.966092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:32.966107 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:32.966157 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:32.966174 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:32.966258 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:32.966459 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:37:32.966534 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:37:32.967530 21576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260927 18:37:32.968089 21576 leader_election.cc:302] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8895f9361cd74b4e938b7fa47893ddc3; no voters:
I20260927 18:37:32.968631 21576 leader_election.cc:288] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:32.968755 21586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:32.969457 21586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [term 1 LEADER]: Becoming Leader. State: Replica: 8895f9361cd74b4e938b7fa47893ddc3, State: Running, Role: LEADER
I20260927 18:37:32.970002 21586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260927 18:37:32.970765 21576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:32.972821 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8895f9361cd74b4e938b7fa47893ddc3. Latest consensus state: current_term: 1 leader_uuid: "8895f9361cd74b4e938b7fa47893ddc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } } }
I20260927 18:37:32.973279 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:32.979455 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:32.984198 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:32.979087 21587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8895f9361cd74b4e938b7fa47893ddc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8895f9361cd74b4e938b7fa47893ddc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } } }
I20260927 18:37:32.986294 21587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:32.994568 21593 catalog_manager.cc:1347] Generated new cluster ID: 4112779ad8724b92ab4a993abe022776
I20260927 18:37:32.994690 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:33.009133 21593 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:33.010911 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:33.020381 21593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8895f9361cd74b4e938b7fa47893ddc3: Generated new TSK 0
I20260927 18:37:33.021277 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:33.034010 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21509
W20260927 18:37:33.035188 21571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.225:53429
W20260927 18:37:33.035948 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21509: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:33.036098 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21509
W20260927 18:37:33.053408 21494 connection.cc:535] client connection to 127.0.0.1:45725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45725): Connection reset by peer (error 104)
2026-09-27T18:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:33 dist-test-slave-n3p4 krb5kdc[21607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:33 dist-test-slave-n3p4 krb5kdc[21607](info): set up 2 sockets
Sep 27 18:37:33 dist-test-slave-n3p4 krb5kdc[21607](info): commencing operation
krb5kdc: starting...
W20260927 18:37:34.142196 389 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:34 dist-test-slave-n3p4 krb5kdc[21607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:34.380915 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:34.708931 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:34.709225 21620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:34.709326 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:34.716583 21620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:34.716698 21620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:34.716755 21620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:34.716825 21620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:34.716866 21620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:34.726410 21620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:34.727625 21620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:34.728894 21620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:34.740916 21625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:34.740839 21628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:34.740950 21626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:34.742100 21620 server_base.cc:1034] running on GCE node
I20260927 18:37:34.742771 21620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:34.744074 21620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:34.745311 21620 hybrid_clock.cc:648] HybridClock initialized: now 1790534254745244 us; error 66 us; skew 500 ppm
I20260927 18:37:34.745695 21620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:34 dist-test-slave-n3p4 krb5kdc[21607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:34.752334 21620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:34.753926 21620 webserver.cc:460] Webserver started at http://127.0.0.1:39823/ using document root <none> and password file <none>
I20260927 18:37:34.754609 21620 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:34.754752 21620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:34.755061 21620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:34.757999 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "adf883e8207b45929be5620deb1a734b"
format_stamp: "Formatted at 2026-09-27 18:37:34 on dist-test-slave-n3p4"
server_key: "6c1d2f68f470905946beeaadbaae45b7"
server_key_iv: "02b249b06582079916a109ab434176a9"
server_key_version: "encryptionkey@0"
I20260927 18:37:34.758740 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "adf883e8207b45929be5620deb1a734b"
format_stamp: "Formatted at 2026-09-27 18:37:34 on dist-test-slave-n3p4"
server_key: "6c1d2f68f470905946beeaadbaae45b7"
server_key_iv: "02b249b06582079916a109ab434176a9"
server_key_version: "encryptionkey@0"
I20260927 18:37:34.763749 21620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:37:34.767257 21635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:34.768595 21620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:37:34.768776 21620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "adf883e8207b45929be5620deb1a734b"
format_stamp: "Formatted at 2026-09-27 18:37:34 on dist-test-slave-n3p4"
server_key: "6c1d2f68f470905946beeaadbaae45b7"
server_key_iv: "02b249b06582079916a109ab434176a9"
server_key_version: "encryptionkey@0"
I20260927 18:37:34.768975 21620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:34.774698 21620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:34.778869 21620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:34.779156 21620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:34.797667 21620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40149
I20260927 18:37:34.797684 21686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40149 every 8 connection(s)
I20260927 18:37:34.799084 21620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:34.802709 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21620
I20260927 18:37:34.802860 21687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:34.802937 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:34.803368 389 external_mini_cluster.cc:1428] Setting key 46370542de5aba736c94c08790846f9d
I20260927 18:37:34.811484 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:34.805057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:39919 (local address 127.0.0.1:40149)
0927 18:37:34.805872 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:34.805888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:34.805931 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:37:34.807635 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:34.807654 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:34.807975 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:34.808390 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:34.809467 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:34.809478 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:34.809496 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:34.809526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:34.809625 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:34.809655 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:34.809900 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:34.810419 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 18:37:34.810599 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20260927 18:37:34.815114 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:34.816677 21687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b: Bootstrap starting.
I20260927 18:37:34.821130 21687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:34.822633 21687 log.cc:824] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:34.823437 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:34.820622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:50803 (local address 127.0.0.1:40149)
0927 18:37:34.820761 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:34.820765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:34.821327 (+ 562us) server_negotiation.cc:407] Connection header received
0927 18:37:34.821530 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:34.821534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:34.821583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:34.821658 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:34.822678 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:34.822681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:34.822685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:34.822717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:34.822753 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:34.822785 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:34.822908 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:34.823066 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:37:34.823123 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:37:34.825626 21687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b: No bootstrap required, opened a new log
I20260927 18:37:34.833062 21687 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260927 18:37:34.833420 21687 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:34.833499 21687 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf883e8207b45929be5620deb1a734b, State: Initialized, Role: FOLLOWER
I20260927 18:37:34.834082 21687 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260927 18:37:34.834303 21687 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:34.834409 21687 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:34.834559 21687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:34.836766 21687 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260927 18:37:34.837348 21687 leader_election.cc:302] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf883e8207b45929be5620deb1a734b; no voters:
I20260927 18:37:34.838073 21687 leader_election.cc:288] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:34.838135 21697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:34.838943 21697 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [term 1 LEADER]: Becoming Leader. State: Replica: adf883e8207b45929be5620deb1a734b, State: Running, Role: LEADER
I20260927 18:37:34.839433 21697 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260927 18:37:34.840345 21687 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:34.842121 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf883e8207b45929be5620deb1a734b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260927 18:37:34.842559 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:34.843420 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:34.843240 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf883e8207b45929be5620deb1a734b. Latest consensus state: current_term: 1 leader_uuid: "adf883e8207b45929be5620deb1a734b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf883e8207b45929be5620deb1a734b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260927 18:37:34.843577 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:34.849395 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:34.859970 21704 catalog_manager.cc:1347] Generated new cluster ID: 5701c25458a841919821e02c617e110d
I20260927 18:37:34.860088 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:34.889160 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:34.889590 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:34.898276 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf883e8207b45929be5620deb1a734b: Generated new TSK 0
I20260927 18:37:34.899000 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:34.912410 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21620
W20260927 18:37:34.913516 21682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.225:39919
W20260927 18:37:34.914309 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21620: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:34.914487 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21620
2026-09-27T18:37:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:35 dist-test-slave-n3p4 krb5kdc[21718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:35 dist-test-slave-n3p4 krb5kdc[21718](info): set up 2 sockets
Sep 27 18:37:35 dist-test-slave-n3p4 krb5kdc[21718](info): commencing operation
krb5kdc: starting...
W20260927 18:37:36.018306 389 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:36.261582 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:36.583112 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:36.583539 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:36.583737 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:36.591678 21731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:36.591832 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:36.591952 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:36.592068 21731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:36.592163 21731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:36.602681 21731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:36.603915 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:36.605212 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:36.616580 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:36.616642 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:36.616549 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:36.617344 21731 server_base.cc:1034] running on GCE node
I20260927 18:37:36.618136 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:36.619408 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:36.620671 21731 hybrid_clock.cc:648] HybridClock initialized: now 1790534256620610 us; error 45 us; skew 500 ppm
I20260927 18:37:36.621059 21731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:36.624063 21731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:36.625443 21731 webserver.cc:460] Webserver started at http://127.0.0.1:44135/ using document root <none> and password file <none>
I20260927 18:37:36.626143 21731 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:36.626282 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:36.626586 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:36.629462 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "7709283e7040425faadac48dd588c5c9"
format_stamp: "Formatted at 2026-09-27 18:37:36 on dist-test-slave-n3p4"
server_key: "40a755a589ee4216777f615169a668d7"
server_key_iv: "bb58c2f86f886f6f5694278022971bb8"
server_key_version: "encryptionkey@0"
I20260927 18:37:36.630199 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "7709283e7040425faadac48dd588c5c9"
format_stamp: "Formatted at 2026-09-27 18:37:36 on dist-test-slave-n3p4"
server_key: "40a755a589ee4216777f615169a668d7"
server_key_iv: "bb58c2f86f886f6f5694278022971bb8"
server_key_version: "encryptionkey@0"
I20260927 18:37:36.635071 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 18:37:36.638391 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:36.639565 21731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:36.639742 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "7709283e7040425faadac48dd588c5c9"
format_stamp: "Formatted at 2026-09-27 18:37:36 on dist-test-slave-n3p4"
server_key: "40a755a589ee4216777f615169a668d7"
server_key_iv: "bb58c2f86f886f6f5694278022971bb8"
server_key_version: "encryptionkey@0"
I20260927 18:37:36.639940 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:36.660274 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:36.664181 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:36.664472 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:36.683192 21731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45985
I20260927 18:37:36.683228 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45985 every 8 connection(s)
I20260927 18:37:36.684535 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:36.688452 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:36.694203 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21731
I20260927 18:37:36.694397 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:36.694823 389 external_mini_cluster.cc:1428] Setting key 6a8d7f8fa3c4683c5d554b7b438c42fd
I20260927 18:37:36.700225 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9: Bootstrap starting.
I20260927 18:37:36.703243 21798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:36.704201 21798 log.cc:824] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:36.707898 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9: No bootstrap required, opened a new log
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:37:36.717015 21798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260927 18:37:36.717494 21798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:36.717614 21798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7709283e7040425faadac48dd588c5c9, State: Initialized, Role: FOLLOWER
I20260927 18:37:36.718253 21798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260927 18:37:36.718520 21798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:36.718564 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:36.696766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:39747 (local address 127.0.0.1:45985)
0927 18:37:36.697561 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:36.697576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:36.697617 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:37:36.699376 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:36.699397 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:36.699777 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:36.700156 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:36.701557 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:36.702929 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:36.704297 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:36.705366 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:36.705536 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:36.710641 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:36.710649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:36.710662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:36.710703 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:36.713978 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:36.714898 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:36.714904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:36.714911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:36.715088 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:36.715593 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:36.715596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:36.715598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:36.716162 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:36.716393 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:36.716887 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:37:36.717221 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":330,"threads_started":1}
I20260927 18:37:36.718672 21798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:36.718858 21798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:36.721294 21798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260927 18:37:36.721853 21798 leader_election.cc:302] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7709283e7040425faadac48dd588c5c9; no voters:
I20260927 18:37:36.722430 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:36.722522 21798 leader_election.cc:288] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:36.722623 21803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:36.723701 21803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 7709283e7040425faadac48dd588c5c9, State: Running, Role: LEADER
I20260927 18:37:36.724327 21803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260927 18:37:36.724802 21798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:36.727164 21805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7709283e7040425faadac48dd588c5c9. Latest consensus state: current_term: 1 leader_uuid: "7709283e7040425faadac48dd588c5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } } }
I20260927 18:37:36.727604 21805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:36.728643 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7709283e7040425faadac48dd588c5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709283e7040425faadac48dd588c5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } } }
I20260927 18:37:36.728988 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:36.739025 21813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:36.746951 21813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:36.755095 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:36.728348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:60931 (local address 127.0.0.1:45985)
0927 18:37:36.733372 (+ 5024us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:36.733382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:36.737541 (+ 4159us) server_negotiation.cc:407] Connection header received
0927 18:37:36.738072 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:36.738077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:36.738160 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:36.738239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:36.740319 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:36.741127 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:36.742425 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:36.743234 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:36.743354 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:36.746800 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:36.746809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:36.746815 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:36.746863 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:36.750880 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:36.751977 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:36.751983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:36.751987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:36.752129 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:36.752705 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:36.752711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:36.752715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:36.753167 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:36.753436 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:36.754541 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0927 18:37:36.754719 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4857}
I20260927 18:37:36.761116 21813 catalog_manager.cc:1347] Generated new cluster ID: eb42ffc29ec94a418a696be8fa498b9c
I20260927 18:37:36.761215 21813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:36.770107 21813 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:36.771334 21813 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:36.777026 21813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7709283e7040425faadac48dd588c5c9: Generated new TSK 0
I20260927 18:37:36.777889 21813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:36.783159 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21731
I20260927 18:37:36.784428 21793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:36.878700 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21731
W20260927 18:37:36.896167 21716 connection.cc:535] client connection to 127.0.0.1:45985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45985): Connection reset by peer (error 104)
2026-09-27T18:37:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21830](info): set up 2 sockets
Sep 27 18:37:36 dist-test-slave-n3p4 krb5kdc[21830](info): commencing operation
krb5kdc: starting...
W20260927 18:37:38.002358 389 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:38 dist-test-slave-n3p4 krb5kdc[21830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:38.240813 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:38.565466 21843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:38.565760 21843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:38.565958 21843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:38.573482 21843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:38.573633 21843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:38.573766 21843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:38.573936 21843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:38.574012 21843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:38.584416 21843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:38.585813 21843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:38.587181 21843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:38.599246 21849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:38.599254 21848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:38.599341 21851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:38.600175 21843 server_base.cc:1034] running on GCE node
I20260927 18:37:38.600998 21843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:38.602519 21843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:38.603753 21843 hybrid_clock.cc:648] HybridClock initialized: now 1790534258603722 us; error 56 us; skew 500 ppm
I20260927 18:37:38.604126 21843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:38 dist-test-slave-n3p4 krb5kdc[21830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:38.607240 21843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:38.608657 21843 webserver.cc:460] Webserver started at http://127.0.0.1:46455/ using document root <none> and password file <none>
I20260927 18:37:38.609360 21843 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:38.609498 21843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:38.609881 21843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:38.612798 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "c7a08826d4b0448bbf1bcc5f024040b3"
format_stamp: "Formatted at 2026-09-27 18:37:38 on dist-test-slave-n3p4"
server_key: "e28affafdfbc52414ccedec5ed81a6e5"
server_key_iv: "d1ac1f7528d5ef7a6950079f41847510"
server_key_version: "encryptionkey@0"
I20260927 18:37:38.613528 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "c7a08826d4b0448bbf1bcc5f024040b3"
format_stamp: "Formatted at 2026-09-27 18:37:38 on dist-test-slave-n3p4"
server_key: "e28affafdfbc52414ccedec5ed81a6e5"
server_key_iv: "d1ac1f7528d5ef7a6950079f41847510"
server_key_version: "encryptionkey@0"
I20260927 18:37:38.618760 21843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 18:37:38.621982 21858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:38.623236 21843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:37:38.623428 21843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "c7a08826d4b0448bbf1bcc5f024040b3"
format_stamp: "Formatted at 2026-09-27 18:37:38 on dist-test-slave-n3p4"
server_key: "e28affafdfbc52414ccedec5ed81a6e5"
server_key_iv: "d1ac1f7528d5ef7a6950079f41847510"
server_key_version: "encryptionkey@0"
I20260927 18:37:38.623610 21843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:38.636096 21843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:38.640174 21843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:38.640451 21843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:38.658421 21843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45907
I20260927 18:37:38.658468 21909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45907 every 8 connection(s)
I20260927 18:37:38.659777 21843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:38.663393 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:38.663512 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21843
I20260927 18:37:38.663697 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:38.664122 389 external_mini_cluster.cc:1428] Setting key c8a0d585f596786b66e4f4efc7ab8ccf
I20260927 18:37:38.676538 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3: Bootstrap starting.
I20260927 18:37:38.677093 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:38.665820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:42317 (local address 127.0.0.1:45907)
0927 18:37:38.666634 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:38.666645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:38.666686 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:37:38.668279 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:38.668300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:38.668678 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:38.669004 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:38.669618 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:38.670967 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:38.672757 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:38.673367 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:38.673489 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:38.674751 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:38.674758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:38.674773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:38.675313 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:38.675384 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:38.675406 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:38.675519 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:38.675826 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 18:37:38.676001 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":365,"threads_started":1}
I20260927 18:37:38.680169 21910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:38.681090 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:38.681423 21910 log.cc:824] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:38.684396 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3: No bootstrap required, opened a new log
I20260927 18:37:38.692428 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:38.685931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:51773 (local address 127.0.0.1:45907)
0927 18:37:38.686102 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:38.686108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:38.686615 (+ 507us) server_negotiation.cc:407] Connection header received
0927 18:37:38.686743 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:38.686748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:38.686820 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:38.686936 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:38.687324 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:38.688450 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:38.689735 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:38.690461 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:38.690584 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:38.691648 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:38.691655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:38.691661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:38.691680 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:38.691721 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:38.691740 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:38.691864 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:38.692015 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:37:38.692081 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:37:38.693728 21910 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260927 18:37:38.694100 21910 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:38.694171 21910 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a08826d4b0448bbf1bcc5f024040b3, State: Initialized, Role: FOLLOWER
I20260927 18:37:38.694615 21910 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260927 18:37:38.694801 21910 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:38.694898 21910 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:38.695014 21910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:38.697000 21910 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260927 18:37:38.697449 21910 leader_election.cc:302] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a08826d4b0448bbf1bcc5f024040b3; no voters:
I20260927 18:37:38.698136 21910 leader_election.cc:288] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:38.698235 21920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:38.699005 21920 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [term 1 LEADER]: Becoming Leader. State: Replica: c7a08826d4b0448bbf1bcc5f024040b3, State: Running, Role: LEADER
I20260927 18:37:38.699569 21920 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260927 18:37:38.700244 21910 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:38.702131 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } } }
I20260927 18:37:38.702440 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:38.703266 21926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:38.703141 21922 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7a08826d4b0448bbf1bcc5f024040b3. Latest consensus state: current_term: 1 leader_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a08826d4b0448bbf1bcc5f024040b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } } }
I20260927 18:37:38.703501 21922 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:38.709095 21926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:38.718194 21926 catalog_manager.cc:1347] Generated new cluster ID: 2fb9a1b510504ceebcbf482dd6c863fc
I20260927 18:37:38.718322 21926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:38.744053 21926 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:38.745354 21926 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:38.759861 21926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7a08826d4b0448bbf1bcc5f024040b3: Generated new TSK 0
I20260927 18:37:38.760797 21926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:38.776378 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21843
W20260927 18:37:38.777552 21905 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.225:42317
W20260927 18:37:38.778313 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21843: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:38.778463 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21843
W20260927 18:37:38.795408 21828 connection.cc:535] client connection to 127.0.0.1:45907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45907): Connection reset by peer (error 104)
2026-09-27T18:37:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:38 dist-test-slave-n3p4 krb5kdc[21941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:38 dist-test-slave-n3p4 krb5kdc[21941](info): set up 2 sockets
Sep 27 18:37:38 dist-test-slave-n3p4 krb5kdc[21941](info): commencing operation
krb5kdc: starting...
W20260927 18:37:39.898226 389 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:40 dist-test-slave-n3p4 krb5kdc[21941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:37:40.142578 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:40.463807 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:40.464073 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:40.464162 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:40.471079 21954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:40.471180 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:40.471226 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:40.471287 21954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:40.471330 21954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:40.480625 21954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:40.481906 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:40.483085 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:40.494462 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:40.494500 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:40.494535 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:40.495585 21954 server_base.cc:1034] running on GCE node
I20260927 18:37:40.496245 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:40.497502 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:40.498692 21954 hybrid_clock.cc:648] HybridClock initialized: now 1790534260498661 us; error 28 us; skew 500 ppm
I20260927 18:37:40.498986 21954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:40 dist-test-slave-n3p4 krb5kdc[21941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:40.501942 21954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:37:40.503278 21954 webserver.cc:460] Webserver started at http://127.0.0.1:36503/ using document root <none> and password file <none>
I20260927 18:37:40.503950 21954 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:40.504074 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:40.504395 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:40.507313 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "5eb12ea031e54d7fa34c001f662d7aa7"
format_stamp: "Formatted at 2026-09-27 18:37:40 on dist-test-slave-n3p4"
server_key: "c7080998262bcc60e5dce6ddbc0abeb3"
server_key_iv: "c3a9beb49490cbc7e57b47dc51516e19"
server_key_version: "encryptionkey@0"
I20260927 18:37:40.508044 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "5eb12ea031e54d7fa34c001f662d7aa7"
format_stamp: "Formatted at 2026-09-27 18:37:40 on dist-test-slave-n3p4"
server_key: "c7080998262bcc60e5dce6ddbc0abeb3"
server_key_iv: "c3a9beb49490cbc7e57b47dc51516e19"
server_key_version: "encryptionkey@0"
I20260927 18:37:40.512832 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:37:40.516163 21969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:40.517480 21954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:40.517652 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "5eb12ea031e54d7fa34c001f662d7aa7"
format_stamp: "Formatted at 2026-09-27 18:37:40 on dist-test-slave-n3p4"
server_key: "c7080998262bcc60e5dce6ddbc0abeb3"
server_key_iv: "c3a9beb49490cbc7e57b47dc51516e19"
server_key_version: "encryptionkey@0"
I20260927 18:37:40.517884 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:40.523304 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:40.527221 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:40.527513 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:40.546049 21954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40901
I20260927 18:37:40.546094 22020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40901 every 8 connection(s)
I20260927 18:37:40.547448 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:40.550995 22021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:40.554641 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 21954
I20260927 18:37:40.554826 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:40.555335 389 external_mini_cluster.cc:1428] Setting key ed2223b20c01e64acff6ccf796209499
I20260927 18:37:40.563925 22021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7: Bootstrap starting.
I20260927 18:37:40.567332 22021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:40.568483 22021 log.cc:824] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:40.568943 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:40.562572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:53141 (local address 127.0.0.1:40901)
0927 18:37:40.563445 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:40.563457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:40.563503 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:37:40.565402 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:40.565424 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:40.565853 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:40.566191 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:40.567234 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:40.567241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:40.567256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:40.567271 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:40.567330 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:40.567357 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:40.567576 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:40.567693 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:37:40.567920 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":402,"threads_started":1}
I20260927 18:37:40.571918 22021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7: No bootstrap required, opened a new log
I20260927 18:37:40.572947 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:40.579594 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:40.577127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:33895 (local address 127.0.0.1:40901)
0927 18:37:40.577309 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:40.577316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:40.577662 (+ 346us) server_negotiation.cc:407] Connection header received
0927 18:37:40.577864 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:40.577868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:40.577920 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:40.578021 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:40.578956 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:40.578966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:40.578969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:40.578983 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:40.579020 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:40.579034 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:40.579174 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:40.579278 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:37:40.579346 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:37:40.580399 22021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260927 18:37:40.580897 22021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:40.581020 22021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb12ea031e54d7fa34c001f662d7aa7, State: Initialized, Role: FOLLOWER
I20260927 18:37:40.581593 22021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260927 18:37:40.581874 22021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:40.582022 22021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:40.582201 22021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:40.584463 22021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260927 18:37:40.584944 22021 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb12ea031e54d7fa34c001f662d7aa7; no voters:
I20260927 18:37:40.585526 22021 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:40.585654 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:40.586553 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [term 1 LEADER]: Becoming Leader. State: Replica: 5eb12ea031e54d7fa34c001f662d7aa7, State: Running, Role: LEADER
I20260927 18:37:40.587047 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260927 18:37:40.587646 22021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:40.589560 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb12ea031e54d7fa34c001f662d7aa7. Latest consensus state: current_term: 1 leader_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260927 18:37:40.590003 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:40.590641 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb12ea031e54d7fa34c001f662d7aa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260927 18:37:40.590953 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:40.592293 22037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:40.597211 22037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:40.606688 22037 catalog_manager.cc:1347] Generated new cluster ID: f893bb612a724eb0bb5adcc74cc04eec
I20260927 18:37:40.606825 22037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:40.628870 22037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:40.629391 22037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:40.638842 22037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eb12ea031e54d7fa34c001f662d7aa7: Generated new TSK 0
I20260927 18:37:40.639884 22037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:40.646100 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21954
W20260927 18:37:40.647512 22016 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.225:53141
W20260927 18:37:40.648285 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 21954: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:40.648504 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 21954
2026-09-27T18:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:40 dist-test-slave-n3p4 krb5kdc[22052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:40 dist-test-slave-n3p4 krb5kdc[22052](info): set up 2 sockets
Sep 27 18:37:40 dist-test-slave-n3p4 krb5kdc[22052](info): commencing operation
krb5kdc: starting...
W20260927 18:37:41.738174 389 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:41 dist-test-slave-n3p4 krb5kdc[22052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:41.982182 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33757
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:42.303584 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:42.303860 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:42.304023 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:42.311259 22065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:42.311375 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:42.311508 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:42.311607 22065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:42.311673 22065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:42.322181 22065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:42.323520 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:42.324831 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:42.336488 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:42.336467 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:42.336732 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:42.337420 22065 server_base.cc:1034] running on GCE node
I20260927 18:37:42.338292 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:42.339700 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:42.340943 22065 hybrid_clock.cc:648] HybridClock initialized: now 1790534262340891 us; error 40 us; skew 500 ppm
I20260927 18:37:42.341313 22065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:42 dist-test-slave-n3p4 krb5kdc[22052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:42.344381 22065 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:42.345692 22065 webserver.cc:460] Webserver started at http://127.0.97.126:33745/ using document root <none> and password file <none>
I20260927 18:37:42.346381 22065 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:42.346529 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:42.346875 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:42.349641 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "0113509882a7450ba53d3b9fba4b540f"
format_stamp: "Formatted at 2026-09-27 18:37:42 on dist-test-slave-n3p4"
server_key: "44ce72a4b2f257c656890ba97cd11537"
server_key_iv: "309fdbcd34f206e88863418fbb12d85f"
server_key_version: "encryptionkey@0"
I20260927 18:37:42.350456 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "0113509882a7450ba53d3b9fba4b540f"
format_stamp: "Formatted at 2026-09-27 18:37:42 on dist-test-slave-n3p4"
server_key: "44ce72a4b2f257c656890ba97cd11537"
server_key_iv: "309fdbcd34f206e88863418fbb12d85f"
server_key_version: "encryptionkey@0"
I20260927 18:37:42.355568 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:37:42.358971 22080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:42.360193 22065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:37:42.360369 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "0113509882a7450ba53d3b9fba4b540f"
format_stamp: "Formatted at 2026-09-27 18:37:42 on dist-test-slave-n3p4"
server_key: "44ce72a4b2f257c656890ba97cd11537"
server_key_iv: "309fdbcd34f206e88863418fbb12d85f"
server_key_version: "encryptionkey@0"
I20260927 18:37:42.360556 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:42.379618 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:42.383430 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:42.383699 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:42.401616 22065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33757
I20260927 18:37:42.401654 22131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33757 every 8 connection(s)
I20260927 18:37:42.402951 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:42.403595 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22065
I20260927 18:37:42.403823 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:42.404419 389 external_mini_cluster.cc:1428] Setting key 6ee4588e98d87dec7ca3218356fb3f1d
I20260927 18:37:42.407274 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:37:42 dist-test-slave-n3p4 krb5kdc[22052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:42.420322 22132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f: Bootstrap starting.
I20260927 18:37:42.423563 22132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:42.424595 22132 log.cc:824] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:42.427091 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.406277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50105 (local address 127.0.97.126:33757)
0927 18:37:42.407072 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:42.407083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:42.407121 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:37:42.408699 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:42.408719 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:42.409080 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:42.409376 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:42.411146 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:42.412488 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:42.414243 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:42.414902 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:42.415022 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:42.419975 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:42.419988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:42.420003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:42.420038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:42.423103 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:42.423891 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:42.423898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:42.423904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:42.424074 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:42.424526 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:42.424529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:42.424531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:42.425135 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:42.425492 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:42.425914 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 18:37:42.426172 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":371,"threads_started":1}
I20260927 18:37:42.427645 22132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f: No bootstrap required, opened a new log
I20260927 18:37:42.431169 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:42.436503 22132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } }
I20260927 18:37:42.436935 22132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:42.437040 22132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0113509882a7450ba53d3b9fba4b540f, State: Initialized, Role: FOLLOWER
I20260927 18:37:42.437602 22132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } }
I20260927 18:37:42.437875 22132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:42.438062 22132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:42.438252 22132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:42.440411 22132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } }
I20260927 18:37:42.440928 22132 leader_election.cc:302] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0113509882a7450ba53d3b9fba4b540f; no voters:
I20260927 18:37:42.441601 22132 leader_election.cc:288] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:42.441709 22142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:42.443584 22132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:42.444435 22142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [term 1 LEADER]: Becoming Leader. State: Replica: 0113509882a7450ba53d3b9fba4b540f, State: Running, Role: LEADER
I20260927 18:37:42.445145 22142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } }
I20260927 18:37:42.451100 22143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0113509882a7450ba53d3b9fba4b540f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } } }
I20260927 18:37:42.451087 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0113509882a7450ba53d3b9fba4b540f. Latest consensus state: current_term: 1 leader_uuid: "0113509882a7450ba53d3b9fba4b540f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0113509882a7450ba53d3b9fba4b540f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33757 } } }
I20260927 18:37:42.451526 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:42.451526 22143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:42.452492 22153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:42.453879 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.435304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57355 (local address 127.0.97.126:33757)
0927 18:37:42.435499 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:42.435508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:42.435885 (+ 377us) server_negotiation.cc:407] Connection header received
0927 18:37:42.436072 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:42.436078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:42.436145 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:42.436245 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:42.437838 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:42.438834 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:42.440063 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:42.440532 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:42.440666 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:42.444181 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:42.444187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:42.444192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:42.444231 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:42.448923 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:42.450212 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:42.450220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:42.450226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:42.450485 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:42.451214 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:42.451222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:42.451228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:42.451991 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:42.452488 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:42.453219 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 18:37:42.453427 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:37:42.458451 22153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:42.467033 22153 catalog_manager.cc:1347] Generated new cluster ID: 6078cf89a66d4e30ac55f343c75664dc
I20260927 18:37:42.467150 22153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:42.473585 22153 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:42.475344 22153 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:42.483735 22153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0113509882a7450ba53d3b9fba4b540f: Generated new TSK 0
I20260927 18:37:42.484509 22153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:42.491074 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22065
I20260927 18:37:42.492254 22127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:42.580806 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22065
W20260927 18:37:42.596015 22050 connection.cc:535] client connection to 127.0.97.126:33757 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:33757): Connection reset by peer (error 104)
2026-09-27T18:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:42 dist-test-slave-n3p4 krb5kdc[22164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:42 dist-test-slave-n3p4 krb5kdc[22164](info): set up 2 sockets
Sep 27 18:37:42 dist-test-slave-n3p4 krb5kdc[22164](info): commencing operation
krb5kdc: starting...
W20260927 18:37:43.678297 389 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:43 dist-test-slave-n3p4 krb5kdc[22164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:43.920728 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42759
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:44.242884 22177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:44.243220 22177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:44.243387 22177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:44.250587 22177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:44.250749 22177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:44.250854 22177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:44.250962 22177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:44.251039 22177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:44.260874 22177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:44.262197 22177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:44.263453 22177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:44.274778 22185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:44.274858 22182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:44.274837 22183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:44.275627 22177 server_base.cc:1034] running on GCE node
I20260927 18:37:44.276413 22177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:44.277741 22177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:44.279012 22177 hybrid_clock.cc:648] HybridClock initialized: now 1790534264278968 us; error 38 us; skew 500 ppm
I20260927 18:37:44.279368 22177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:44 dist-test-slave-n3p4 krb5kdc[22164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:44.282441 22177 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:44.283845 22177 webserver.cc:460] Webserver started at http://127.0.97.126:36561/ using document root <none> and password file <none>
I20260927 18:37:44.284482 22177 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:44.284578 22177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:44.284827 22177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:44.287674 22177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "11c8acde08ca4e7282254578714418ad"
format_stamp: "Formatted at 2026-09-27 18:37:44 on dist-test-slave-n3p4"
server_key: "0748ebc472efa6a04927048d5ab032f0"
server_key_iv: "68ca0cf414786e3e719c02e652af0095"
server_key_version: "encryptionkey@0"
I20260927 18:37:44.288355 22177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "11c8acde08ca4e7282254578714418ad"
format_stamp: "Formatted at 2026-09-27 18:37:44 on dist-test-slave-n3p4"
server_key: "0748ebc472efa6a04927048d5ab032f0"
server_key_iv: "68ca0cf414786e3e719c02e652af0095"
server_key_version: "encryptionkey@0"
I20260927 18:37:44.293648 22177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:37:44.297434 22192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:44.298924 22177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:44.299129 22177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "11c8acde08ca4e7282254578714418ad"
format_stamp: "Formatted at 2026-09-27 18:37:44 on dist-test-slave-n3p4"
server_key: "0748ebc472efa6a04927048d5ab032f0"
server_key_iv: "68ca0cf414786e3e719c02e652af0095"
server_key_version: "encryptionkey@0"
I20260927 18:37:44.299343 22177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:44.311074 22177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:44.315526 22177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:44.315830 22177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:44.335386 22177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42759
I20260927 18:37:44.335397 22243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42759 every 8 connection(s)
I20260927 18:37:44.336856 22177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:44.340488 22244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:44.343796 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22177
I20260927 18:37:44.343973 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:44.344518 389 external_mini_cluster.cc:1428] Setting key 2d62c1ee58c58c8a630d2ea7709a18da
I20260927 18:37:44.352421 22244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad: Bootstrap starting.
I20260927 18:37:44.356021 22244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:44.357142 22244 log.cc:824] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:44.357848 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:44.346245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39467 (local address 127.0.97.126:42759)
0927 18:37:44.347037 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:44.347047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:44.347087 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:37:44.348685 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:44.348706 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:44.349088 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:44.349392 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:44.349968 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:44.351300 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:44.353110 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:44.353888 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:44.354044 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:44.355341 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:44.355349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:44.355363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:44.355915 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:44.356000 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:44.356031 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:44.356235 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:44.356558 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 18:37:44.356710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":388,"threads_started":1}
I20260927 18:37:44.359943 22244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad: No bootstrap required, opened a new log
I20260927 18:37:44.361754 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:44.367964 22244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } }
I20260927 18:37:44.368320 22244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:44.368444 22244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c8acde08ca4e7282254578714418ad, State: Initialized, Role: FOLLOWER
I20260927 18:37:44.369025 22244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } }
I20260927 18:37:44.369266 22244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:44.369407 22244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:44.369582 22244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:44.371796 22244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } }
I20260927 18:37:44.372191 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:44.365931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45551 (local address 127.0.97.126:42759)
0927 18:37:44.366094 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:44.366101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:44.366532 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:37:44.366745 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:44.366748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:44.366811 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:44.366919 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:44.367448 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:44.368159 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:44.369684 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:44.370201 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:44.370342 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:44.371482 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:44.371494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:44.371498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:44.371514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:44.371564 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:44.371579 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:44.371698 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:44.371886 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:37:44.371935 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:37:44.372380 22244 leader_election.cc:302] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c8acde08ca4e7282254578714418ad; no voters:
I20260927 18:37:44.373090 22244 leader_election.cc:288] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:44.373195 22254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:44.373867 22254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [term 1 LEADER]: Becoming Leader. State: Replica: 11c8acde08ca4e7282254578714418ad, State: Running, Role: LEADER
I20260927 18:37:44.374488 22254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } }
I20260927 18:37:44.375675 22244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:44.377183 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c8acde08ca4e7282254578714418ad. Latest consensus state: current_term: 1 leader_uuid: "11c8acde08ca4e7282254578714418ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } } }
I20260927 18:37:44.377648 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:44.377427 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c8acde08ca4e7282254578714418ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c8acde08ca4e7282254578714418ad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42759 } } }
I20260927 18:37:44.377871 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:44.379401 22260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:44.384030 22260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:44.392413 22260 catalog_manager.cc:1347] Generated new cluster ID: a43983b3ae6742a5964eb8cfdd79b99f
I20260927 18:37:44.392518 22260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:44.418502 22260 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:44.419894 22260 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:44.432538 22260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c8acde08ca4e7282254578714418ad: Generated new TSK 0
I20260927 18:37:44.433436 22260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:44.446376 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22177
W20260927 18:37:44.447715 22239 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39467
W20260927 18:37:44.448431 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:44.448576 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22177
W20260927 18:37:44.466030 22162 connection.cc:535] client connection to 127.0.97.126:42759 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:42759): Connection reset by peer (error 104)
2026-09-27T18:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:44 dist-test-slave-n3p4 krb5kdc[22275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:44 dist-test-slave-n3p4 krb5kdc[22275](info): set up 2 sockets
Sep 27 18:37:44 dist-test-slave-n3p4 krb5kdc[22275](info): commencing operation
krb5kdc: starting...
W20260927 18:37:45.546166 389 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:45 dist-test-slave-n3p4 krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:45.786988 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42981
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:46.113919 22288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:46.114239 22288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:46.114405 22288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:46.121346 22288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:46.121493 22288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:46.121596 22288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:46.121701 22288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:46.121811 22288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:46.131433 22288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:46.132666 22288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:46.134083 22288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:46.146085 22296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:46.146131 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:46.146567 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:46.146786 22288 server_base.cc:1034] running on GCE node
I20260927 18:37:46.147567 22288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:46.148963 22288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:46.150211 22288 hybrid_clock.cc:648] HybridClock initialized: now 1790534266150175 us; error 47 us; skew 500 ppm
I20260927 18:37:46.150574 22288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:46 dist-test-slave-n3p4 krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:46.158610 22288 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:46.160092 22288 webserver.cc:460] Webserver started at http://127.0.97.126:44961/ using document root <none> and password file <none>
I20260927 18:37:46.160738 22288 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:46.160835 22288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:46.161089 22288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:46.164031 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "e4738aa61523417d82bd468f5d1c4aa6"
format_stamp: "Formatted at 2026-09-27 18:37:46 on dist-test-slave-n3p4"
server_key: "b5e496ff5b6aaaf03f315c87a15c2bc2"
server_key_iv: "55b53875fae5734eb7cf9c39ca393605"
server_key_version: "encryptionkey@0"
I20260927 18:37:46.164763 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "e4738aa61523417d82bd468f5d1c4aa6"
format_stamp: "Formatted at 2026-09-27 18:37:46 on dist-test-slave-n3p4"
server_key: "b5e496ff5b6aaaf03f315c87a15c2bc2"
server_key_iv: "55b53875fae5734eb7cf9c39ca393605"
server_key_version: "encryptionkey@0"
I20260927 18:37:46.169646 22288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:37:46.172991 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:46.174355 22288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:46.174521 22288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "e4738aa61523417d82bd468f5d1c4aa6"
format_stamp: "Formatted at 2026-09-27 18:37:46 on dist-test-slave-n3p4"
server_key: "b5e496ff5b6aaaf03f315c87a15c2bc2"
server_key_iv: "55b53875fae5734eb7cf9c39ca393605"
server_key_version: "encryptionkey@0"
I20260927 18:37:46.174701 22288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:46.180190 22288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:46.184163 22288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:46.184439 22288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:46.202813 22288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42981
I20260927 18:37:46.202917 22354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42981 every 8 connection(s)
I20260927 18:37:46.204166 22288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:46.207849 22355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:46.209084 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22288
I20260927 18:37:46.209265 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:46.209887 389 external_mini_cluster.cc:1428] Setting key 9fcebcd5714080da151b76ad8b7601e8
I20260927 18:37:46.217448 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:46.211636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48371 (local address 127.0.97.126:42981)
0927 18:37:46.212472 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:46.212483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:46.212522 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:37:46.214170 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:46.214192 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:46.214521 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:46.214838 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:46.215914 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:46.215920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:46.215934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:46.215949 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:46.216010 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:46.216033 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:46.216192 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:46.216311 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:37:46.216502 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":389,"threads_started":1}
I20260927 18:37:46.221378 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:46.221370 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6: Bootstrap starting.
I20260927 18:37:46.225070 22355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:46.226445 22355 log.cc:824] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:46.229189 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:46.226550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57791 (local address 127.0.97.126:42981)
0927 18:37:46.226725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:46.226733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:46.227291 (+ 558us) server_negotiation.cc:407] Connection header received
0927 18:37:46.227473 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:46.227478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:46.227540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:46.227644 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:46.228513 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:46.228518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:46.228523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:46.228540 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:46.228586 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:46.228601 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:46.228714 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:46.228882 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 18:37:46.228939 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:37:46.229339 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6: No bootstrap required, opened a new log
I20260927 18:37:46.236974 22355 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } }
I20260927 18:37:46.237339 22355 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:46.237457 22355 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4738aa61523417d82bd468f5d1c4aa6, State: Initialized, Role: FOLLOWER
I20260927 18:37:46.238089 22355 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } }
I20260927 18:37:46.238317 22355 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:46.238472 22355 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:46.238649 22355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:46.241041 22355 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } }
I20260927 18:37:46.241537 22355 leader_election.cc:302] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4738aa61523417d82bd468f5d1c4aa6; no voters:
I20260927 18:37:46.242120 22355 leader_election.cc:288] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:46.242420 22365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:46.243147 22365 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [term 1 LEADER]: Becoming Leader. State: Replica: e4738aa61523417d82bd468f5d1c4aa6, State: Running, Role: LEADER
I20260927 18:37:46.243683 22365 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } }
I20260927 18:37:46.244218 22355 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:46.246740 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4738aa61523417d82bd468f5d1c4aa6. Latest consensus state: current_term: 1 leader_uuid: "e4738aa61523417d82bd468f5d1c4aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } } }
I20260927 18:37:46.247219 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:46.246711 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4738aa61523417d82bd468f5d1c4aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4738aa61523417d82bd468f5d1c4aa6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42981 } } }
I20260927 18:37:46.247834 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:46.248917 22373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:46.255395 22373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:46.264258 22373 catalog_manager.cc:1347] Generated new cluster ID: 6f3f7a004665401bac4551a924416b4a
I20260927 18:37:46.264405 22373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:46.273509 22373 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:46.273978 22373 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:46.283226 22373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4738aa61523417d82bd468f5d1c4aa6: Generated new TSK 0
I20260927 18:37:46.284137 22373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:46.298327 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22288
W20260927 18:37:46.299969 22350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48371
W20260927 18:37:46.300699 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22288: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:46.300850 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22288
2026-09-27T18:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:46 dist-test-slave-n3p4 krb5kdc[22386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:46 dist-test-slave-n3p4 krb5kdc[22386](info): set up 2 sockets
Sep 27 18:37:46 dist-test-slave-n3p4 krb5kdc[22386](info): commencing operation
krb5kdc: starting...
W20260927 18:37:47.486238 389 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:47 dist-test-slave-n3p4 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:47.730198 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33947
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:48.052088 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:48.052352 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:48.052533 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:48.059497 22399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:48.059597 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:48.059703 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:48.059805 22399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:48.059908 22399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:48.069530 22399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:48.070796 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:48.072094 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:48.084111 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:48.084152 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:48.084820 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:48.085407 22399 server_base.cc:1034] running on GCE node
I20260927 18:37:48.086274 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:48.087687 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:48.088922 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790534268088885 us; error 51 us; skew 500 ppm
I20260927 18:37:48.089262 22399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:48 dist-test-slave-n3p4 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:48.092432 22399 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:48.093876 22399 webserver.cc:460] Webserver started at http://127.0.97.126:35757/ using document root <none> and password file <none>
I20260927 18:37:48.094496 22399 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:48.094589 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:48.094823 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:48.097610 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "121756db7ef44cd29714d5b5faa717dd"
format_stamp: "Formatted at 2026-09-27 18:37:48 on dist-test-slave-n3p4"
server_key: "94c283982a3a014ba586d56eba28d095"
server_key_iv: "a62caa71b0467d12703f97bba2343db5"
server_key_version: "encryptionkey@0"
I20260927 18:37:48.098373 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "121756db7ef44cd29714d5b5faa717dd"
format_stamp: "Formatted at 2026-09-27 18:37:48 on dist-test-slave-n3p4"
server_key: "94c283982a3a014ba586d56eba28d095"
server_key_iv: "a62caa71b0467d12703f97bba2343db5"
server_key_version: "encryptionkey@0"
I20260927 18:37:48.103502 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:37:48.106760 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:48.108098 22399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:48.108300 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "121756db7ef44cd29714d5b5faa717dd"
format_stamp: "Formatted at 2026-09-27 18:37:48 on dist-test-slave-n3p4"
server_key: "94c283982a3a014ba586d56eba28d095"
server_key_iv: "a62caa71b0467d12703f97bba2343db5"
server_key_version: "encryptionkey@0"
I20260927 18:37:48.108510 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:48.124429 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:48.128701 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:48.129014 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:48.147061 22399 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33947
I20260927 18:37:48.147094 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33947 every 8 connection(s)
I20260927 18:37:48.148408 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:48.151986 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:48.153867 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22399
I20260927 18:37:48.154042 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:48.154572 389 external_mini_cluster.cc:1428] Setting key bee8a9b200102b618facff449002fabf
I20260927 18:37:48.166077 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd: Bootstrap starting.
Sep 27 18:37:48 dist-test-slave-n3p4 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:48.169636 22466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:48.170708 22466 log.cc:824] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:48.173377 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd: No bootstrap required, opened a new log
I20260927 18:37:48.176719 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:48.156297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52031 (local address 127.0.97.126:33947)
0927 18:37:48.157081 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:48.157092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:48.157136 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:37:48.158737 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:48.158761 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:48.159157 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:48.159467 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:48.161112 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:48.162473 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:48.164236 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:48.164876 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:48.165010 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:48.169560 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:48.169568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:48.169582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:48.169615 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:48.172771 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:48.173436 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:48.173446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:48.173458 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:48.173654 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:48.174071 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:48.174077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:48.174080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:48.174560 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:48.174899 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:48.175375 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:37:48.175778 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":371,"threads_started":1}
I20260927 18:37:48.180401 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:48.181910 22466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } }
I20260927 18:37:48.182265 22466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:48.182368 22466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121756db7ef44cd29714d5b5faa717dd, State: Initialized, Role: FOLLOWER
I20260927 18:37:48.182948 22466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } }
I20260927 18:37:48.183257 22466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:48.183411 22466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:48.183583 22466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:48.185645 22466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } }
I20260927 18:37:48.186215 22466 leader_election.cc:302] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 121756db7ef44cd29714d5b5faa717dd; no voters:
I20260927 18:37:48.186914 22466 leader_election.cc:288] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:48.187016 22475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:48.188027 22475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [term 1 LEADER]: Becoming Leader. State: Replica: 121756db7ef44cd29714d5b5faa717dd, State: Running, Role: LEADER
I20260927 18:37:48.188779 22475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } }
I20260927 18:37:48.190325 22466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:48.191560 22478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "121756db7ef44cd29714d5b5faa717dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } } }
I20260927 18:37:48.191972 22478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:48.194200 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121756db7ef44cd29714d5b5faa717dd. Latest consensus state: current_term: 1 leader_uuid: "121756db7ef44cd29714d5b5faa717dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121756db7ef44cd29714d5b5faa717dd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33947 } } }
I20260927 18:37:48.194638 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:48.196375 22481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:48.201488 22481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:48.208158 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:48.187399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39981 (local address 127.0.97.126:33947)
0927 18:37:48.187604 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:48.187613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:48.187942 (+ 329us) server_negotiation.cc:407] Connection header received
0927 18:37:48.188213 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:48.188218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:48.188284 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:48.188366 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:48.190158 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:48.191345 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:48.193335 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:48.194329 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:48.194495 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:48.197841 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:48.197852 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:48.197860 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:48.197907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:48.203068 (+ 5161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:48.204162 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:48.204174 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:48.204179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:48.204387 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:48.204957 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:48.204964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:48.204970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:48.206200 (+ 1230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:48.207419 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:48.207574 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 18:37:48.207800 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:37:48.214088 22481 catalog_manager.cc:1347] Generated new cluster ID: 1e7dc8090e64400a87e0061914f32d59
I20260927 18:37:48.214190 22481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:48.230060 22481 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:48.231344 22481 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:48.253891 22481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 121756db7ef44cd29714d5b5faa717dd: Generated new TSK 0
I20260927 18:37:48.254732 22481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:48.260866 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22399
I20260927 18:37:48.261970 22461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:48.353308 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22399
2026-09-27T18:37:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:48 dist-test-slave-n3p4 krb5kdc[22498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:48 dist-test-slave-n3p4 krb5kdc[22498](info): set up 2 sockets
Sep 27 18:37:48 dist-test-slave-n3p4 krb5kdc[22498](info): commencing operation
krb5kdc: starting...
W20260927 18:37:49.462389 389 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:49 dist-test-slave-n3p4 krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:49.708554 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36705
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:50.048094 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:50.048401 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:50.048552 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:50.055725 22511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:50.055848 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:50.055934 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:50.056051 22511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:50.056108 22511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:50.066084 22511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:50.067400 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:50.068696 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:50.080771 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:50.080911 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:50.080978 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:50.081669 22511 server_base.cc:1034] running on GCE node
I20260927 18:37:50.082484 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:50.083868 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:50.085065 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790534270085014 us; error 41 us; skew 500 ppm
I20260927 18:37:50.085422 22511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:50 dist-test-slave-n3p4 krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:50.088567 22511 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:50.090111 22511 webserver.cc:460] Webserver started at http://127.0.97.126:44571/ using document root <none> and password file <none>
I20260927 18:37:50.090883 22511 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:50.091027 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:50.091328 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:50.094270 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "2e9b01d2c8d94a9f85dcf240f98848a2"
format_stamp: "Formatted at 2026-09-27 18:37:50 on dist-test-slave-n3p4"
server_key: "cc25399b567a3a353b444c32622dac00"
server_key_iv: "9cf93439ba0a3766709075b7ffc946bc"
server_key_version: "encryptionkey@0"
I20260927 18:37:50.095052 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "2e9b01d2c8d94a9f85dcf240f98848a2"
format_stamp: "Formatted at 2026-09-27 18:37:50 on dist-test-slave-n3p4"
server_key: "cc25399b567a3a353b444c32622dac00"
server_key_iv: "9cf93439ba0a3766709075b7ffc946bc"
server_key_version: "encryptionkey@0"
I20260927 18:37:50.099969 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:37:50.103420 22526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:50.104660 22511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:37:50.104833 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "2e9b01d2c8d94a9f85dcf240f98848a2"
format_stamp: "Formatted at 2026-09-27 18:37:50 on dist-test-slave-n3p4"
server_key: "cc25399b567a3a353b444c32622dac00"
server_key_iv: "9cf93439ba0a3766709075b7ffc946bc"
server_key_version: "encryptionkey@0"
I20260927 18:37:50.105027 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:50.151197 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:50.155678 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:50.155972 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:50.174572 22511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36705
I20260927 18:37:50.174611 22577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36705 every 8 connection(s)
I20260927 18:37:50.175997 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:50.179699 22578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:50.182273 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22511
I20260927 18:37:50.182451 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:50.182969 389 external_mini_cluster.cc:1428] Setting key e60f13b17c50101f116e66184807862a
I20260927 18:37:50.191422 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2: Bootstrap starting.
I20260927 18:37:50.194329 22578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:50.195461 22578 log.cc:824] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:50.195566 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:50.184793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40415 (local address 127.0.97.126:36705)
0927 18:37:50.185540 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:50.185552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:50.185595 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:37:50.187324 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:50.187345 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:50.187719 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:50.188090 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:50.188660 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:50.190054 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:50.191325 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:50.192023 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:50.192225 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:50.193396 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:50.193404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:50.193416 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:50.193983 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:50.194062 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:50.194085 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:50.194206 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:50.194423 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 18:37:50.194660 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":317,"threads_started":1}
I20260927 18:37:50.198431 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2: No bootstrap required, opened a new log
I20260927 18:37:50.199218 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:50.207439 22578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } }
I20260927 18:37:50.207792 22578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:50.207922 22578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9b01d2c8d94a9f85dcf240f98848a2, State: Initialized, Role: FOLLOWER
I20260927 18:37:50.208617 22578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } }
I20260927 18:37:50.208871 22578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:50.209014 22578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:50.209194 22578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:50.210546 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:50.204020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41033 (local address 127.0.97.126:36705)
0927 18:37:50.204184 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:50.204189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:50.204581 (+ 392us) server_negotiation.cc:407] Connection header received
0927 18:37:50.204710 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:50.204716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:50.204792 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:50.204902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:50.205387 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:50.206521 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:50.207906 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:50.208496 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:50.208636 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:50.209801 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:50.209807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:50.209812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:50.209834 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:50.209901 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:50.209919 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:50.210016 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:50.210138 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:37:50.210240 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:37:50.211967 22578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } }
I20260927 18:37:50.212466 22578 leader_election.cc:302] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e9b01d2c8d94a9f85dcf240f98848a2; no voters:
I20260927 18:37:50.213052 22578 leader_election.cc:288] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:50.213148 22588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:50.213886 22588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2e9b01d2c8d94a9f85dcf240f98848a2, State: Running, Role: LEADER
I20260927 18:37:50.214517 22588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } }
I20260927 18:37:50.215329 22578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:50.217238 22589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } } }
I20260927 18:37:50.217598 22589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:50.217980 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e9b01d2c8d94a9f85dcf240f98848a2. Latest consensus state: current_term: 1 leader_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b01d2c8d94a9f85dcf240f98848a2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36705 } } }
I20260927 18:37:50.219645 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:50.220280 22594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:50.224917 22594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:50.235317 22594 catalog_manager.cc:1347] Generated new cluster ID: 08176b2190a24cca947467120d160e38
I20260927 18:37:50.235450 22594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:50.244415 22594 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:50.246667 22594 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:50.256906 22594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e9b01d2c8d94a9f85dcf240f98848a2: Generated new TSK 0
I20260927 18:37:50.257853 22594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:50.269402 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22511
W20260927 18:37:50.270576 22573 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40415
W20260927 18:37:50.271279 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22511: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:50.271445 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22511
2026-09-27T18:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:50 dist-test-slave-n3p4 krb5kdc[22609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:50 dist-test-slave-n3p4 krb5kdc[22609](info): set up 2 sockets
Sep 27 18:37:50 dist-test-slave-n3p4 krb5kdc[22609](info): commencing operation
krb5kdc: starting...
W20260927 18:37:51.450191 389 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:51 dist-test-slave-n3p4 krb5kdc[22609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:51.699071 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45077
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:52.024272 22622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:52.024555 22622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:52.024721 22622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:52.032015 22622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:52.032127 22622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:52.032217 22622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:52.032302 22622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:52.032382 22622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:52.042169 22622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:52.043433 22622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:52.044711 22622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:52.056435 22630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:52.056530 22628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:52.056795 22627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:52.057533 22622 server_base.cc:1034] running on GCE node
I20260927 18:37:52.058437 22622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:52.059770 22622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:52.061002 22622 hybrid_clock.cc:648] HybridClock initialized: now 1790534272060952 us; error 25 us; skew 500 ppm
I20260927 18:37:52.061380 22622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:52 dist-test-slave-n3p4 krb5kdc[22609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:52.069734 22622 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:52.071259 22622 webserver.cc:460] Webserver started at http://127.0.97.126:33249/ using document root <none> and password file <none>
I20260927 18:37:52.071955 22622 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:52.072093 22622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:52.072417 22622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:52.075407 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "9668181fda3a401b9b1ccb01a54e61d4"
format_stamp: "Formatted at 2026-09-27 18:37:52 on dist-test-slave-n3p4"
server_key: "fe3bf1f608176894556be8d0775b570d"
server_key_iv: "ce77f28d70f7e91778ea3d89066bf7e9"
server_key_version: "encryptionkey@0"
I20260927 18:37:52.076169 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "9668181fda3a401b9b1ccb01a54e61d4"
format_stamp: "Formatted at 2026-09-27 18:37:52 on dist-test-slave-n3p4"
server_key: "fe3bf1f608176894556be8d0775b570d"
server_key_iv: "ce77f28d70f7e91778ea3d89066bf7e9"
server_key_version: "encryptionkey@0"
I20260927 18:37:52.081300 22622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:37:52.084864 22637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:52.086468 22622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:52.086686 22622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "9668181fda3a401b9b1ccb01a54e61d4"
format_stamp: "Formatted at 2026-09-27 18:37:52 on dist-test-slave-n3p4"
server_key: "fe3bf1f608176894556be8d0775b570d"
server_key_iv: "ce77f28d70f7e91778ea3d89066bf7e9"
server_key_version: "encryptionkey@0"
I20260927 18:37:52.086916 22622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:52.093138 22622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:52.097589 22622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:52.097963 22622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:52.116153 22622 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45077
I20260927 18:37:52.116197 22688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45077 every 8 connection(s)
I20260927 18:37:52.117568 22622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:52.121655 22689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:52.122177 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22622
I20260927 18:37:52.122364 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:52.122822 389 external_mini_cluster.cc:1428] Setting key d411dbdc223d42be7f41c2fa5d717d27
I20260927 18:37:52.132242 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:52.124676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53925 (local address 127.0.97.126:45077)
0927 18:37:52.125491 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:52.125511 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:52.125563 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:37:52.128356 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:52.128385 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:52.128827 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:52.129273 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:52.130387 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:52.130395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:52.130411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:52.130427 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:52.130495 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:52.130524 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:52.130736 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:52.130855 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:37:52.131043 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":320,"threads_started":1}
I20260927 18:37:52.135843 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4: Bootstrap starting.
I20260927 18:37:52.136161 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:52.139632 22689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:52.140980 22689 log.cc:824] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:52.143916 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4: No bootstrap required, opened a new log
I20260927 18:37:52.144091 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:52.141254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58497 (local address 127.0.97.126:45077)
0927 18:37:52.141451 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:52.141459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:52.142050 (+ 591us) server_negotiation.cc:407] Connection header received
0927 18:37:52.142229 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:52.142234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:52.142290 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:52.142390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:52.143349 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:52.143360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:52.143364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:52.143378 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:52.143414 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:52.143427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:52.143538 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:52.143678 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 18:37:52.143735 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:37:52.152022 22689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } }
I20260927 18:37:52.152527 22689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:52.152632 22689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9668181fda3a401b9b1ccb01a54e61d4, State: Initialized, Role: FOLLOWER
I20260927 18:37:52.153119 22689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } }
I20260927 18:37:52.153312 22689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:52.153407 22689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:52.153514 22689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:52.155642 22689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } }
I20260927 18:37:52.156222 22689 leader_election.cc:302] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9668181fda3a401b9b1ccb01a54e61d4; no voters:
I20260927 18:37:52.156920 22689 leader_election.cc:288] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:52.157021 22699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:52.157724 22699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [term 1 LEADER]: Becoming Leader. State: Replica: 9668181fda3a401b9b1ccb01a54e61d4, State: Running, Role: LEADER
I20260927 18:37:52.158334 22699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } }
I20260927 18:37:52.159235 22689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:52.161077 22700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9668181fda3a401b9b1ccb01a54e61d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } } }
I20260927 18:37:52.161497 22700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:52.162287 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9668181fda3a401b9b1ccb01a54e61d4. Latest consensus state: current_term: 1 leader_uuid: "9668181fda3a401b9b1ccb01a54e61d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9668181fda3a401b9b1ccb01a54e61d4" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45077 } } }
I20260927 18:37:52.162668 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:52.162997 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:52.169361 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:52.180519 22705 catalog_manager.cc:1347] Generated new cluster ID: afe2708db83b41069a94f4ad4b151cc2
I20260927 18:37:52.180645 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:52.206136 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:52.206666 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:52.221972 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9668181fda3a401b9b1ccb01a54e61d4: Generated new TSK 0
I20260927 18:37:52.223078 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:52.231902 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22622
W20260927 18:37:52.233366 22684 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53925
W20260927 18:37:52.234046 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22622: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:37:52.234195 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22622
2026-09-27T18:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1962 ms)
[----------] 18 tests from AuthTokenIssuingTest (34697 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:52 dist-test-slave-n3p4 krb5kdc[22720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:52 dist-test-slave-n3p4 krb5kdc[22720](info): set up 2 sockets
Sep 27 18:37:52 dist-test-slave-n3p4 krb5kdc[22720](info): commencing operation
krb5kdc: starting...
W20260927 18:37:53.334470 389 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:53 dist-test-slave-n3p4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:53.589857 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44611
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:53.911778 22733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:53.912104 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:53.912176 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:53.912328 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:53.919735 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:53.919873 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:53.920002 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:53.920086 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:53.920192 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:53.930063 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:53.931389 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:53.932696 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:53.944276 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:53.944792 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:53.945998 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:53.946731 22733 server_base.cc:1034] running on GCE node
I20260927 18:37:53.947607 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:53.949060 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:53.950297 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790534273950232 us; error 50 us; skew 500 ppm
I20260927 18:37:53.950664 22733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:53 dist-test-slave-n3p4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:53.953959 22733 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:53.955379 22733 webserver.cc:460] Webserver started at http://127.0.97.126:38985/ using document root <none> and password file <none>
I20260927 18:37:53.956084 22733 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:53.956209 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:53.956529 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:53.959501 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "468bbacd6c9447aa8dd64eb4e99f6871"
format_stamp: "Formatted at 2026-09-27 18:37:53 on dist-test-slave-n3p4"
server_key: "7acf972ac772a6699785e03d665e582f"
server_key_iv: "de5373b576397419d9c1222d2f5a9e2f"
server_key_version: "encryptionkey@0"
I20260927 18:37:53.960255 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "468bbacd6c9447aa8dd64eb4e99f6871"
format_stamp: "Formatted at 2026-09-27 18:37:53 on dist-test-slave-n3p4"
server_key: "7acf972ac772a6699785e03d665e582f"
server_key_iv: "de5373b576397419d9c1222d2f5a9e2f"
server_key_version: "encryptionkey@0"
I20260927 18:37:53.965149 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:37:53.968433 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:53.969767 22733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:37:53.970031 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "468bbacd6c9447aa8dd64eb4e99f6871"
format_stamp: "Formatted at 2026-09-27 18:37:53 on dist-test-slave-n3p4"
server_key: "7acf972ac772a6699785e03d665e582f"
server_key_iv: "de5373b576397419d9c1222d2f5a9e2f"
server_key_version: "encryptionkey@0"
I20260927 18:37:53.970271 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:53.988277 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:53.992583 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:53.992847 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:54.011356 22733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44611
I20260927 18:37:54.011379 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44611 every 8 connection(s)
I20260927 18:37:54.012756 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:54.013391 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22733
I20260927 18:37:54.013609 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:54.014380 389 external_mini_cluster.cc:1428] Setting key 50e5bd00ed588c43bdafca174c747205
I20260927 18:37:54.016690 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:54.023155 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:54.049984 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.026280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.0.97.126:44611)
0927 18:37:54.028259 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:54.028264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:54.028309 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:37:54.030572 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:54.030578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:54.030799 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:54.030990 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:54.032662 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.034062 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.036340 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.036975 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.037120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.042024 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:54.042037 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:54.042056 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:54.042102 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:54.045957 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.046638 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.046649 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.046655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.046848 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.047220 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.047227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.047233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.047955 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:54.048163 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:54.048643 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 18:37:54.049059 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":1790,"spinlock_wait_cycles":14720,"thread_start_us":1546,"threads_started":1}
I20260927 18:37:54.050058 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.026280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44611 (local address 127.0.0.1:48652)
0927 18:37:54.027258 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.027306 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:54.030169 (+ 2863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:54.031074 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:54.031082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:54.031904 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:54.032425 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.032453 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.034220 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.034227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.036186 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.036201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.037245 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.037254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.037472 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.038247 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:54.038279 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:54.041727 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:54.046086 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.046096 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.046104 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.046497 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.046957 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.046960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.046962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.047113 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.048095 (+ 982us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:54.048101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:54.048282 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 18:37:54.048486 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:37:54.048857 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":462,"threads_started":1}
I20260927 18:37:54.052946 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:54.054172 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.053736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38224)
0927 18:37:54.053963 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.054062 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 18:37:54.054432 22752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
I20260927 18:37:54.056123 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.055673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38236)
0927 18:37:54.055849 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.055947 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 18:37:54.056409 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:43421: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260927 18:37:54.073032 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:43421
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:54.105755 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } attempt: 1
I20260927 18:37:54.107012 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.106560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38246)
0927 18:37:54.106749 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.106868 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 18:37:54.107985 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.107643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38254)
0927 18:37:54.107801 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.107865 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:37:54.108158 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:43421: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
I20260927 18:37:54.176386 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } attempt: 2
I20260927 18:37:54.177510 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.177022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38268)
0927 18:37:54.177274 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.177374 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:37:54.178660 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.178306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38276)
0927 18:37:54.178454 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.178531 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:37:54.178887 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:43421: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
I20260927 18:37:54.270186 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } attempt: 3
I20260927 18:37:54.271404 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.270955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38292)
0927 18:37:54.271155 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.271255 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 18:37:54.272544 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.272169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:38294)
0927 18:37:54.272338 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.272412 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 18:37:54.272719 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:43421: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
W20260927 18:37:54.396047 22807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:54.396410 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:54.396541 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:54.396689 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:54.403721 22807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:54.403834 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:54.403930 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:54.404027 22807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:54.404114 22807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:54.413327 22807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:43421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:54.414687 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:54.415940 22807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:54.415997 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } attempt: 4
I20260927 18:37:54.417397 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.416801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:58126)
0927 18:37:54.417149 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.417248 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 18:37:54.418607 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.418182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:58140)
0927 18:37:54.418347 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.418443 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 18:37:54.418896 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:43421: Network error: Client connection negotiation failed: client connection to 127.0.97.125:43421: connect: Connection refused (error 111)
W20260927 18:37:54.427736 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:54.427732 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:54.427696 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:54.428649 22807 server_base.cc:1034] running on GCE node
I20260927 18:37:54.429404 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:54.430754 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:54.431995 22807 hybrid_clock.cc:648] HybridClock initialized: now 1790534274431936 us; error 42 us; skew 500 ppm
I20260927 18:37:54.432312 22807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:54.435452 22807 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260927 18:37:54.436945 22807 webserver.cc:460] Webserver started at http://127.0.97.125:40161/ using document root <none> and password file <none>
I20260927 18:37:54.437613 22807 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:54.437745 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:54.438093 22807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:54.440946 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data/instance:
uuid: "bc053a2e6a4749048f8e970ba37a7fd7"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "266c0f944c60921a536812aa95850989"
server_key_iv: "98612758ef44ff27bb3ed9922e445623"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.441712 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal/instance:
uuid: "bc053a2e6a4749048f8e970ba37a7fd7"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "266c0f944c60921a536812aa95850989"
server_key_iv: "98612758ef44ff27bb3ed9922e445623"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.446781 22807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:37:54.450302 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:54.451601 22807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:37:54.451774 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "bc053a2e6a4749048f8e970ba37a7fd7"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "266c0f944c60921a536812aa95850989"
server_key_iv: "98612758ef44ff27bb3ed9922e445623"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.451990 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:54.470815 22807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:54.475263 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:54.475554 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:54.493185 22807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:43421
I20260927 18:37:54.493222 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:43421 every 8 connection(s)
I20260927 18:37:54.494549 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
I20260927 18:37:54.495868 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22807
I20260927 18:37:54.496047 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-1/wal/instance
I20260927 18:37:54.496498 389 external_mini_cluster.cc:1428] Setting key 0c4625be664ab83079423880bfaf23a3
I20260927 18:37:54.498508 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:54.504971 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
I20260927 18:37:54.525210 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.507647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.97.126:44611)
0927 18:37:54.507827 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:54.507832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:54.509171 (+ 1339us) server_negotiation.cc:407] Connection header received
0927 18:37:54.511112 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:54.511117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:54.511210 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:54.511320 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:54.512794 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.513476 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.515126 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.515540 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.515680 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.519503 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:54.519508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:54.519511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:54.519545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:54.522273 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.522977 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.522987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.522990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.523101 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.523518 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.523523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.523526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.523796 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:54.523967 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:54.524663 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 18:37:54.524888 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:37:54.526275 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.508219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44611 (local address 127.0.0.1:35262)
0927 18:37:54.509004 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.509043 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:54.510837 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:54.511560 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:54.511568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:54.512071 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:54.512624 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.512642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.513649 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.513653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.514970 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.514983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.515764 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.515771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.516018 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.516747 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:54.516771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:54.519276 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:54.522413 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.522421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.522438 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.522846 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.523208 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.523214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.523217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.523402 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.523944 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:54.523951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:54.524297 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 18:37:54.524565 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 18:37:54.525054 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260927 18:37:54.528007 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:54.544236 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.528860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:58154)
0927 18:37:54.529058 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.529082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:54.529201 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:54.530070 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:54.530073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:54.530414 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:54.530548 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.530555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.532577 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.532580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.533392 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.533401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.534583 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.534587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.534707 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.535262 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:54.535278 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:54.537394 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:54.541689 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.541693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.541695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.542087 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.542567 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.542573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.542576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.542729 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.543512 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:54.543515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:54.543652 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 18:37:54.543764 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:37:54.543975 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:37:54.544737 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.528870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.97.125:43421)
0927 18:37:54.529530 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:54.529540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:54.529574 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:37:54.529628 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:54.529637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:54.529918 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:54.530078 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:54.530718 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.532451 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.533523 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.534455 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.534587 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.537607 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:54.537622 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:54.537641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:54.537683 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:54.541561 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.542209 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.542221 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.542227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.542395 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.542847 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.542855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.542859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.543350 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:54.543579 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:54.543974 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 18:37:54.544268 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":275,"threads_started":1}
I20260927 18:37:54.547538 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:54.548606 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.548177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47240)
0927 18:37:54.548358 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.548465 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:37:54.548894 22825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.550557 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.550232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47250)
0927 18:37:54.550374 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.550453 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:37:54.550805 22874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260927 18:37:54.557207 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:38333
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:54.600106 22874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 1
I20260927 18:37:54.601259 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.600826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47264)
0927 18:37:54.601015 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.601118 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 18:37:54.602304 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.601976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47266)
0927 18:37:54.602126 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.602208 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:37:54.602500 22874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.670714 22874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 2
I20260927 18:37:54.671850 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.671397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47276)
0927 18:37:54.671594 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.671694 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:37:54.672955 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.672562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47290)
0927 18:37:54.672752 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.672834 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:37:54.673179 22874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.718158 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } attempt: 5
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:54.732059 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.719115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.97.125:43421)
0927 18:37:54.719308 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:54.719314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:54.719336 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:37:54.719406 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:54.719411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:54.719497 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:54.719605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:54.720409 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.721127 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.722388 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.722823 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.722969 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.726501 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:54.726518 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:54.726522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:54.726553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:54.729295 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.730202 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.730207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.730209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.730327 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:54.730752 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:54.730755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:54.730757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:54.731051 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:54.731160 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:54.731644 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 18:37:54.731811 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:54.732156 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.718918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:58164)
0927 18:37:54.719117 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.719144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:54.719291 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:54.719652 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:54.719657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:54.720085 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:54.720254 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.720262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.721276 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.721281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.722232 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:54.722244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:54.722981 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:54.722987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:54.723194 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:54.723833 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:54.723852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:54.726303 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:54.729460 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.729467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.729472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.730075 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.730442 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:54.730447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:54.730450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:54.730628 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:54.731172 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:54.731177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:54.731355 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 18:37:54.731528 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 18:37:54.731806 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:37:54.733218 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:54.734306 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.733848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47302)
0927 18:37:54.734042 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.734137 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:37:54.735255 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.734899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47316)
0927 18:37:54.735050 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.735128 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:37:54.735471 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.764387 22874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 3
I20260927 18:37:54.765543 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.765071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47326)
0927 18:37:54.765290 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.765393 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 18:37:54.766789 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.766388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47332)
0927 18:37:54.766617 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.766687 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260927 18:37:54.767038 22874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.786692 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 1
I20260927 18:37:54.787961 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.787352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47346)
0927 18:37:54.787596 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.787723 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:37:54.789018 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.788579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47352)
0927 18:37:54.788755 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.788849 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:37:54.789225 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.857451 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 2
I20260927 18:37:54.858788 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.858151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47362)
0927 18:37:54.858410 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.858564 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:37:54.859885 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.859486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47370)
0927 18:37:54.859651 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.859736 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:37:54.860088 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
W20260927 18:37:54.890206 22881 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:54.890558 22881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:54.890666 22881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:54.890789 22881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:54.897965 22881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:54.898051 22881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:54.898149 22881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:54.898222 22881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:54.898308 22881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:54.907919 22881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44611,127.0.97.125:43421,127.0.97.124:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:54.909219 22881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:54.910245 22874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 4
I20260927 18:37:54.910601 22881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:54.911386 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.910931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47378)
0927 18:37:54.911140 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.911226 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:37:54.912293 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.912015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47394)
0927 18:37:54.912143 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.912204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:37:54.912559 22874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
W20260927 18:37:54.922814 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:54.922851 22889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:54.922777 22887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:54.923648 22881 server_base.cc:1034] running on GCE node
I20260927 18:37:54.924371 22881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:54.925698 22881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:54.926941 22881 hybrid_clock.cc:648] HybridClock initialized: now 1790534274926885 us; error 29 us; skew 500 ppm
I20260927 18:37:54.927261 22881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:54 dist-test-slave-n3p4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:54.930488 22881 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260927 18:37:54.931900 22881 webserver.cc:460] Webserver started at http://127.0.97.124:45683/ using document root <none> and password file <none>
I20260927 18:37:54.932520 22881 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:54.932605 22881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:54.932864 22881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:54.935681 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data/instance:
uuid: "3ae37f010a374cf5ab87b25bb0ce27b3"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "6663f033568764922311cc69a62976ff"
server_key_iv: "5bfe7fb418edf76482acd61a53df920a"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.936383 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal/instance:
uuid: "3ae37f010a374cf5ab87b25bb0ce27b3"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "6663f033568764922311cc69a62976ff"
server_key_iv: "5bfe7fb418edf76482acd61a53df920a"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.942516 22881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:37:54.945757 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:54.947165 22881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:37:54.947295 22881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "3ae37f010a374cf5ab87b25bb0ce27b3"
format_stamp: "Formatted at 2026-09-27 18:37:54 on dist-test-slave-n3p4"
server_key: "6663f033568764922311cc69a62976ff"
server_key_iv: "5bfe7fb418edf76482acd61a53df920a"
server_key_version: "encryptionkey@0"
I20260927 18:37:54.947468 22881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:54.966290 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 3
I20260927 18:37:54.967648 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.966991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47410)
0927 18:37:54.967266 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.967406 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:37:54.968814 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.968401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47424)
0927 18:37:54.968620 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.968704 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 18:37:54.969091 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:38333: Network error: Client connection negotiation failed: client connection to 127.0.97.124:38333: connect: Connection refused (error 111)
I20260927 18:37:54.974800 22881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:54.979240 22881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:54.979511 22881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:54.996618 22881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:38333
I20260927 18:37:54.996635 22947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:38333 every 8 connection(s)
I20260927 18:37:54.997956 22881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
I20260927 18:37:55.001543 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 22881
I20260927 18:37:55.001614 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:55.001739 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790533919012461-389-0/minicluster-data/master-2/wal/instance
I20260927 18:37:55.002214 389 external_mini_cluster.cc:1428] Setting key 4c49da197cad4eb8093be6438c035cd5
I20260927 18:37:55.010998 22948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:55.026381 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.003922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.97.124:38333)
0927 18:37:55.004708 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.004724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.004767 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:37:55.006424 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.006446 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.006826 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.007159 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.008942 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.010251 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.012108 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.013066 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.013248 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.018923 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.018933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.018946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.018994 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.022347 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.023134 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.023140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.023145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.023316 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.023736 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.023740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.023742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.024240 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.024538 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.024992 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.025375 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":367,"threads_started":1}
I20260927 18:37:55.029595 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.013938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44611 (local address 127.0.0.1:35276)
0927 18:37:55.014761 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.014825 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:55.015073 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:55.015482 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:55.015491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:55.016362 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:55.016706 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.016719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.018239 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.018247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.019539 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.019551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.020493 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.020500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.020687 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.021227 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:55.021244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:55.023758 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:55.027020 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.027030 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.027037 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.027423 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.027802 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.027806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.027810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.027953 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.028460 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:55.028467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:55.028663 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 18:37:55.028870 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 18:37:55.029172 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":404,"threads_started":1}
I20260927 18:37:55.029615 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.013907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.0.97.126:44611)
0927 18:37:55.014095 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.014101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.014927 (+ 826us) server_negotiation.cc:407] Connection header received
0927 18:37:55.015220 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.015227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.015300 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.015399 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.016883 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.018061 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.019688 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.020285 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.020446 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.023945 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.023949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.023952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.023985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.026890 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.027544 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.027550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.027558 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.027687 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.028060 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.028066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.028069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.028314 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.028502 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.028971 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.029209 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:37:55.031298 22948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:55.044160 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.029326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.97.126:44611)
0927 18:37:55.029906 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.029911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.029928 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:37:55.029984 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.029988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.030056 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.030167 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.031418 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.032436 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.033596 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.034079 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.034213 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.038178 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.038184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.038188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.038250 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.041561 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.042201 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.042206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.042209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.042371 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.042742 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.042748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.042752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.043023 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.043291 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.043573 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.043781 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":479,"threads_started":1}
I20260927 18:37:55.044675 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.032025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.0.97.125:43421)
0927 18:37:55.032202 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.032208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.032295 (+ 87us) server_negotiation.cc:407] Connection header received
0927 18:37:55.032434 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.032437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.032495 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.032602 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.033254 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.034003 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.035277 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.035676 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.035808 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.038948 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.038955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.038961 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.038999 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.042245 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.042830 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.042837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.042840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.042993 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.043354 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.043383 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.043385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.043658 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.043822 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.044230 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.044431 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:37:55.044703 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.032014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43421 (local address 127.0.0.1:58168)
0927 18:37:55.032210 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.032228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:55.032323 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:55.032621 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:55.032625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:55.032987 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:55.033119 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.033127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.034137 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.034142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.035138 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.035149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.035818 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.035823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.035997 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.036497 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:55.036514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:55.038714 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:55.042375 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.042378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.042380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.042710 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.043104 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.043110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.043114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.043249 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.043787 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:55.043791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:55.043954 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 18:37:55.044105 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:37:55.044384 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:37:55.045881 22948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:55.059038 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.046551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47432)
0927 18:37:55.046686 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.046703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:55.046790 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:55.047109 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:55.047113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:55.047465 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:55.047587 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.047595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.048525 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.048530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.049338 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.049348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.050086 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.050091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.050283 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.050925 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:55.050950 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:55.053205 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:55.056543 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.056550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.056554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.056982 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.057306 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.057313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.057318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.057450 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.058009 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:55.058014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:55.058159 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 18:37:55.058310 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:37:55.058617 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 18:37:55.059085 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.046753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.0.97.124:38333)
0927 18:37:55.046892 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.046897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.046911 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:37:55.046953 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.046957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.047002 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.047096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.047716 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.048388 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.049476 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.049939 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.050072 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.053362 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.053372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.053376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.053401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.056417 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.057089 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.057096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.057099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.057206 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.057547 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.057555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.057558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.057897 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.058035 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.058393 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.058770 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":49}
I20260927 18:37:55.060590 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.046208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.97.125:43421)
0927 18:37:55.046365 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.046371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.046389 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:37:55.046560 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.046566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.046634 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.046732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.047861 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.048560 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.049725 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.050197 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.050322 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.054566 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.054571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.054574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.054602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.057414 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.058191 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.058198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.058201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.058340 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.058716 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.058723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.058726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.058985 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.059255 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.060071 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.060293 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:37:55.064646 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: Bootstrap starting.
I20260927 18:37:55.065340 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:55.067962 22948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:55.069018 22948 log.cc:824] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:55.067857 22775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "468bbacd6c9447aa8dd64eb4e99f6871"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35280
I20260927 18:37:55.071795 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: No bootstrap required, opened a new log
I20260927 18:37:55.082144 22948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.082571 22948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae37f010a374cf5ab87b25bb0ce27b3, State: Initialized, Role: FOLLOWER
I20260927 18:37:55.083067 22948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.083920 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.084244 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.085031 22948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:37:55.094611 22966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:37:55.094738 22966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:37:55.092509 22881 master_runner.cc:186] Error getting master registration for 127.0.97.126:44611: OK, instance_id { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" instance_seqno: 1790534274006083 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:37:55.103668 22881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:37:55.146327 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 4
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260927 18:37:55.163213 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.147166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47442)
0927 18:37:55.147368 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.147390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:55.147559 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:55.147954 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:55.147959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:55.148453 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:55.148681 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.148692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.150237 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.150244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.151732 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.151750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.152762 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.152769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.152986 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.153666 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:55.153684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:55.156084 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:55.160357 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.160363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.160366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.160910 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.161306 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.161311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.161315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.161501 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.162190 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:55.162197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:55.162364 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 18:37:55.162508 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 18:37:55.162795 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:37:55.163239 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.147364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.0.97.124:38333)
0927 18:37:55.147579 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.147586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.147609 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:55.147702 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.147708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.147786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.147913 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.148845 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.150072 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.151875 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.152604 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.152722 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.156363 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.156370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.156377 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.156428 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.160201 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.161030 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.161038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.161048 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.161187 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.161611 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.161616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.161620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.162046 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.162210 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.162639 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.162863 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:37:55.169623 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: Bootstrap starting.
I20260927 18:37:55.173696 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:55.175056 22800 log.cc:824] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:55.178553 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: No bootstrap required, opened a new log
I20260927 18:37:55.186132 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.186573 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468bbacd6c9447aa8dd64eb4e99f6871, State: Initialized, Role: FOLLOWER
I20260927 18:37:55.187072 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.188021 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.188469 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.189234 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:55.189324 22775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:37:55.189518 22775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:55.193522 22775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.195597 22775 leader_election.cc:288] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [CANDIDATE]: Term 1 election: Requested vote from peers bc053a2e6a4749048f8e970ba37a7fd7 (127.0.97.125:43421), 3ae37f010a374cf5ab87b25bb0ce27b3 (127.0.97.124:38333)
I20260927 18:37:55.196833 22923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3"
I20260927 18:37:55.197299 22923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:55.200870 22923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468bbacd6c9447aa8dd64eb4e99f6871 in term 1.
I20260927 18:37:55.201537 22751 leader_election.cc:302] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae37f010a374cf5ab87b25bb0ce27b3, 468bbacd6c9447aa8dd64eb4e99f6871; no voters:
I20260927 18:37:55.202924 22969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 18:37:55.204144 22980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:37:55.204449 22980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:37:55.211733 22874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } attempt: 5
Sep 27 18:37:55 dist-test-slave-n3p4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
W20260927 18:37:55.231504 22902 tablet.cc:2369] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:55.242350 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.212440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:38333 (local address 127.0.0.1:47456)
0927 18:37:55.212856 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.212880 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:55.213015 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:55.214123 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:55.214131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:55.214765 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:55.215016 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.215030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.216940 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.216949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.218851 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:55.218869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.220501 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.220510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:55.220822 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.221749 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:55.221818 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:55.226086 (+ 4268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:55.234130 (+ 8044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.234140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.234147 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.235284 (+ 1137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.236282 (+ 998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:55.236289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:55.236322 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:55.236673 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:55.238106 (+ 1433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:55.238139 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:55.238475 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 18:37:55.238746 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 18:37:55.239400 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
W20260927 18:37:55.245734 22754 tablet.cc:2369] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:55.247860 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.213274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.97.124:38333)
0927 18:37:55.213515 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.213523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.213544 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:37:55.213623 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.213630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.213725 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.213865 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.215226 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.216723 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.219094 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.220042 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.220180 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.226458 (+ 6278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.226468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.226476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.226547 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.233818 (+ 7271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.235646 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.235685 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.235691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.236037 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.236964 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.236972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.237011 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.237862 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.240305 (+ 2443us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.241272 (+ 967us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.243557 (+ 2285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:37:55.256592 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: Bootstrap starting.
I20260927 18:37:55.261719 22874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:55.269165 22969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [term 1 LEADER]: Becoming Leader. State: Replica: 468bbacd6c9447aa8dd64eb4e99f6871, State: Running, Role: LEADER
I20260927 18:37:55.269860 22969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.270229 22874 log.cc:824] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:55.298367 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468bbacd6c9447aa8dd64eb4e99f6871. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.300060 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:55.300450 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: No bootstrap required, opened a new log
I20260927 18:37:55.302513 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.275029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.0.97.126:44611)
0927 18:37:55.275255 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.275264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.281622 (+ 6358us) server_negotiation.cc:407] Connection header received
0927 18:37:55.282151 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.282157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.282243 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.282325 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.284696 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.285843 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.287753 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.288502 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.288606 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.292035 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.292042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.292047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.292083 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.296565 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.299819 (+ 3254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.299825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.299839 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.300034 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.300546 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.300552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.300555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.300948 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.301203 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.301816 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.302098 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:37:55.285658 22733 master_runner.cc:186] Error getting master registration for 127.0.97.125:43421: OK, instance_id { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" instance_seqno: 1790534274488101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:37:55.315976 22733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:37:55.318378 22991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:55.320680 22991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:55.324018 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.305416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.0.97.124:38333)
0927 18:37:55.305625 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.305631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.306072 (+ 441us) server_negotiation.cc:407] Connection header received
0927 18:37:55.306354 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.306362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.306452 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.306552 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.308354 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.309432 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.311317 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.312002 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.312101 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.316663 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.316671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.316676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.316717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.320586 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.321530 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.321536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.321540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.321689 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.322170 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.322176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.322179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.322550 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.322802 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.323368 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.323601 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:37:55.328577 22874 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.329003 22874 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc053a2e6a4749048f8e970ba37a7fd7, State: Initialized, Role: FOLLOWER
I20260927 18:37:55.329533 22874 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } }
I20260927 18:37:55.336880 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.277107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.0.97.125:43421)
0927 18:37:55.277306 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.277314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.302321 (+ 25007us) server_negotiation.cc:407] Connection header received
0927 18:37:55.302611 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.302619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.302702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.302779 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.307511 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.308623 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.324308 (+ 15685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.324999 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.325105 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.329995 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:55.330002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:55.330008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:55.330055 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:55.333637 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.334465 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.334471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.334475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.334643 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:55.335128 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:55.335141 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:55.335145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:55.335507 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:55.335742 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.336275 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.336514 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:37:55.347235 22923 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [term 1 FOLLOWER]: Refusing update from remote peer 468bbacd6c9447aa8dd64eb4e99f6871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:55.349220 22982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:55.350389 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.350904 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.351692 22874 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:55.354242 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc053a2e6a4749048f8e970ba37a7fd7"
I20260927 18:37:55.354789 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:55.365713 22849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468bbacd6c9447aa8dd64eb4e99f6871 in term 1.
I20260927 18:37:55.366806 22848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [term 1 FOLLOWER]: Refusing update from remote peer 468bbacd6c9447aa8dd64eb4e99f6871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:55.369274 22982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:55.414042 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468bbacd6c9447aa8dd64eb4e99f6871. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.414513 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.416803 22992 mvcc.cc:205] Tried to move back new op lower bound from 7334028391809589248 to 7334028391512121344. Current Snapshot: MvccSnapshot[applied={T|T < 7334028391809589248}]
I20260927 18:37:55.422297 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.422703 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:55.424474 22996 mvcc.cc:205] Tried to move back new op lower bound from 7334028391809589248 to 7334028391512121344. Current Snapshot: MvccSnapshot[applied={T|T < 7334028391809589248}]
I20260927 18:37:55.431784 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468bbacd6c9447aa8dd64eb4e99f6871. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.432231 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.434574 22991 catalog_manager.cc:1347] Generated new cluster ID: 8553b3fde2074629ba68aaa65b78988c
I20260927 18:37:55.434690 22991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:55.436545 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.436767 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.444186 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.444475 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:55.450049 22995 mvcc.cc:205] Tried to move back new op lower bound from 7334028391809589248 to 7334028391512121344. Current Snapshot: MvccSnapshot[applied={T|T < 7334028391809589248}]
I20260927 18:37:55.451121 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468bbacd6c9447aa8dd64eb4e99f6871" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44611 } } peers { permanent_uuid: "bc053a2e6a4749048f8e970ba37a7fd7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43421 } } peers { permanent_uuid: "3ae37f010a374cf5ab87b25bb0ce27b3" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 38333 } } }
I20260927 18:37:55.451380 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:55.473855 22991 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:55.475647 22991 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:55.501021 23012 catalog_manager.cc:1259] Loaded cluster ID: 8553b3fde2074629ba68aaa65b78988c
I20260927 18:37:55.501132 23012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: loading cluster ID for follower catalog manager: success
I20260927 18:37:55.504650 22991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 468bbacd6c9447aa8dd64eb4e99f6871: Generated new TSK 0
I20260927 18:37:55.505734 22991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:55.506930 23012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: acquiring CA information for follower catalog manager: success
I20260927 18:37:55.509742 23012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc053a2e6a4749048f8e970ba37a7fd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:37:55.630882 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.618102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.0.97.126:44611)
0927 18:37:55.618350 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.618357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.628080 (+ 9723us) server_negotiation.cc:407] Connection header received
0927 18:37:55.629186 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.629197 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.629286 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.629379 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.630610 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35306: BlockingRecv error: recv got EOF from 127.0.0.1:35306 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 18:37:55.635649 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.621969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.97.125:43421)
0927 18:37:55.622155 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.622161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.632288 (+ 10127us) server_negotiation.cc:407] Connection header received
0927 18:37:55.634267 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.634273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.634388 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.634469 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.635414 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58198: BlockingRecv error: recv got EOF from 127.0.0.1:58198 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:37:55.637619 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.633209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.0.97.124:38333)
0927 18:37:55.633382 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.633390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.634009 (+ 619us) server_negotiation.cc:407] Connection header received
0927 18:37:55.636153 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.636158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.636243 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.636325 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.637361 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47472: BlockingRecv error: recv got EOF from 127.0.0.1:47472 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:37:55.686374 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.670493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.97.125:43421)
0927 18:37:55.670697 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.670704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.677722 (+ 7018us) server_negotiation.cc:407] Connection header received
0927 18:37:55.678336 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.678345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.678572 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.678743 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.679578 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.680715 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.682935 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.683691 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.683822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.684694 (+ 872us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.685411 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.685547 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.685859 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.685966 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:37:55.731104 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.721444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.0.97.124:38333)
0927 18:37:55.721680 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.721687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.729032 (+ 7345us) server_negotiation.cc:407] Connection header received
0927 18:37:55.729547 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.729553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.729694 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.729770 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.730868 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47476: BlockingRecv error: recv got EOF from 127.0.0.1:47476 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:37:55.747314 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.741373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.97.125:43421)
0927 18:37:55.741602 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.741611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.742167 (+ 556us) server_negotiation.cc:407] Connection header received
0927 18:37:55.742392 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.742398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.742506 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.742609 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.743056 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.743919 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.745461 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.745952 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.746098 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.746527 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.746658 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.746766 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.746910 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.746992 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:37:55.759557 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.754298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.0.97.124:38333)
0927 18:37:55.754522 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.754531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.758249 (+ 3718us) server_negotiation.cc:407] Connection header received
0927 18:37:55.758438 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.758444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.758527 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.758632 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.759330 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47482: BlockingRecv error: recv got EOF from 127.0.0.1:47482 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:37:55.776324 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.770619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.97.125:43421)
0927 18:37:55.770859 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.770867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.771374 (+ 507us) server_negotiation.cc:407] Connection header received
0927 18:37:55.771576 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.771582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.771662 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.771760 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.772144 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.772908 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.774533 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.774976 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.775107 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.775513 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.775644 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.775741 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.775894 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.775970 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 18:37:55.784192 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.782013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.97.124:38333)
0927 18:37:55.782176 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.782183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.782911 (+ 728us) server_negotiation.cc:407] Connection header received
0927 18:37:55.783112 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.783124 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.783195 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.783309 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.783997 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47490: BlockingRecv error: recv got EOF from 127.0.0.1:47490 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:37:55.802775 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.796392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.125:43421)
0927 18:37:55.796676 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.796685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.797222 (+ 537us) server_negotiation.cc:407] Connection header received
0927 18:37:55.797438 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.797444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.797524 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.797617 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.798082 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.798975 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.801018 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.801498 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.801634 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.802088 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.802188 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.802295 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.802401 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.802489 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260927 18:37:55.813933 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.811658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.97.124:38333)
0927 18:37:55.811979 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.811988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.812506 (+ 518us) server_negotiation.cc:407] Connection header received
0927 18:37:55.812702 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.812708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.812812 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.812918 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.813674 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47494: BlockingRecv error: recv got EOF from 127.0.0.1:47494 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260927 18:37:55.830287 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.824865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.97.125:43421)
0927 18:37:55.825056 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.825062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.825744 (+ 682us) server_negotiation.cc:407] Connection header received
0927 18:37:55.825995 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.826004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.826085 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.826185 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.826562 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.827265 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.828553 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.829004 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.829133 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.829546 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.829647 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.829744 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.829898 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.829979 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 18:37:55.838728 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.836627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.97.124:38333)
0927 18:37:55.836852 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.836858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.837239 (+ 381us) server_negotiation.cc:407] Connection header received
0927 18:37:55.837410 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.837413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.837484 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.837567 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.838462 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47502: BlockingRecv error: recv got EOF from 127.0.0.1:47502 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:37:55.863617 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.858169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.97.125:43421)
0927 18:37:55.858378 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.858385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.858783 (+ 398us) server_negotiation.cc:407] Connection header received
0927 18:37:55.858984 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.858995 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.859081 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.859180 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.859608 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.860394 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.861811 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.862230 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.862455 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.862920 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.863020 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.863094 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.863259 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.863325 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:37:55.871897 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.870092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.0.97.124:38333)
0927 18:37:55.870287 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.870295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.870604 (+ 309us) server_negotiation.cc:407] Connection header received
0927 18:37:55.870732 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.870739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.870834 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.870961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.871635 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47504: BlockingRecv error: recv got EOF from 127.0.0.1:47504 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:37:55.913626 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.907562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.97.125:43421)
0927 18:37:55.907800 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.907810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.908092 (+ 282us) server_negotiation.cc:407] Connection header received
0927 18:37:55.908290 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.908294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.908395 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.908474 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.908989 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.909739 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.911722 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.912185 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.912273 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.912787 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.912893 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.912958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.913234 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.913293 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 18:37:55.921420 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:55.919587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.97.124:38333)
0927 18:37:55.919782 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.919787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.920101 (+ 314us) server_negotiation.cc:407] Connection header received
0927 18:37:55.920221 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.920227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.920335 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.920453 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:55.921182 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47516: BlockingRecv error: recv got EOF from 127.0.0.1:47516 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:37:55.996290 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.989937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.97.125:43421)
0927 18:37:55.990178 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:55.990188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:55.990700 (+ 512us) server_negotiation.cc:407] Connection header received
0927 18:37:55.990925 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:55.990930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:55.991004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:55.991097 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:55.991548 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.992339 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.994487 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:55.994962 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:55.995114 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:55.995548 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:55.995652 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:55.995721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:55.995926 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 18:37:55.995990 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 18:37:56.004161 22951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:56.002373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.97.124:38333)
0927 18:37:56.002551 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.002559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.002838 (+ 279us) server_negotiation.cc:407] Connection header received
0927 18:37:56.002907 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.002911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.002988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.003076 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:56.003937 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47526: BlockingRecv error: recv got EOF from 127.0.0.1:47526 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:37:56.096681 22966 catalog_manager.cc:1259] Loaded cluster ID: 8553b3fde2074629ba68aaa65b78988c
I20260927 18:37:56.096782 22966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: loading cluster ID for follower catalog manager: success
I20260927 18:37:56.099228 22966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: acquiring CA information for follower catalog manager: success
I20260927 18:37:56.100881 22966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ae37f010a374cf5ab87b25bb0ce27b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:37:56.141058 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.136335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.0.97.125:43421)
0927 18:37:56.136571 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.136577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.136779 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:37:56.136947 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.136952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.137031 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.137118 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:56.137524 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.138272 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.139513 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.139901 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.139987 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:56.140320 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:56.140433 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:56.140497 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:56.140687 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:37:56.140747 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 18:37:56.157295 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.151358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.97.124:38333)
0927 18:37:56.151560 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.151568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.151980 (+ 412us) server_negotiation.cc:407] Connection header received
0927 18:37:56.152208 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.152214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.152382 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.152500 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:56.152949 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.153632 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.154994 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.155398 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.155547 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:56.156109 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:56.156669 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:56.156791 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:56.157005 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:37:56.157073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:56.172945 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.163831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.97.125:43421)
0927 18:37:56.164052 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.164060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.165657 (+ 1597us) server_negotiation.cc:407] Connection header received
0927 18:37:56.165921 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.165930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.166050 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.166164 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:56.166790 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.168262 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.170839 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.171505 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.171634 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:56.172210 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:56.172347 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:56.172448 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:56.172608 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:37:56.172686 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:37:56.177151 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.164655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.0.97.124:38333)
0927 18:37:56.164907 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.164915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.171021 (+ 6106us) server_negotiation.cc:407] Connection header received
0927 18:37:56.171074 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.171078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.171134 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.171233 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:56.172003 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.172955 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.175567 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.175979 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.176070 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:56.176480 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:56.176569 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:56.176666 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:56.176788 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:37:56.176860 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:37:56.178295 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.163695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.0.97.126:44611)
0927 18:37:56.170727 (+ 7032us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:56.170731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:56.170746 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:37:56.170882 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:56.170888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:56.171075 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:56.171514 (+ 439us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:56.172076 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.173171 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.176155 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:56.176568 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:56.176712 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:56.177206 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:56.177603 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:56.177712 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:56.177901 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:37:56.177970 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6922,"thread_start_us":6801,"threads_started":1}
I20260927 18:37:56.182993 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22733
I20260927 18:37:56.183936 22795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:56.295274 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22733
I20260927 18:37:56.316674 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22807
I20260927 18:37:56.317873 22869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:56.437860 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22807
I20260927 18:37:56.458396 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 22881
I20260927 18:37:56.459548 22943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:37:56.567545 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 22881
2026-09-27T18:37:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4339 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:56 dist-test-slave-n3p4 krb5kdc[23125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:56 dist-test-slave-n3p4 krb5kdc[23125](info): set up 2 sockets
Sep 27 18:37:56 dist-test-slave-n3p4 krb5kdc[23125](info): commencing operation
krb5kdc: starting...
W20260927 18:37:57.690403 389 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/test-user.keytab.
Sep 27 18:37:57 dist-test-slave-n3p4 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260927 18:37:57.940440 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44345
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:58.262404 23138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:58.262750 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.262902 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.263069 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:58.270174 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:58.270289 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.270396 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.270483 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:58.270571 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:58.280109 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:58.281385 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.282701 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:58.294821 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.294777 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.294920 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.295950 23138 server_base.cc:1034] running on GCE node
I20260927 18:37:58.296769 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.298338 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.299614 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790534278299546 us; error 36 us; skew 500 ppm
I20260927 18:37:58.300021 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:58 dist-test-slave-n3p4 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.307009 23138 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260927 18:37:58.308431 23138 webserver.cc:460] Webserver started at http://127.0.97.126:33955/ using document root <none> and password file <none>
I20260927 18:37:58.309162 23138 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.309319 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.309638 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.312615 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data/instance:
uuid: "e81291caf6204990adb081cee9a987f2"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "2e9f19d2d63d481ea5304ccf8d900a03"
server_key_iv: "33fa82a80c2def883f35d692812dcdc5"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.313391 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance:
uuid: "e81291caf6204990adb081cee9a987f2"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "2e9f19d2d63d481ea5304ccf8d900a03"
server_key_iv: "33fa82a80c2def883f35d692812dcdc5"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.318884 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 18:37:58.322490 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.323872 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:58.324064 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal
uuid: "e81291caf6204990adb081cee9a987f2"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "2e9f19d2d63d481ea5304ccf8d900a03"
server_key_iv: "33fa82a80c2def883f35d692812dcdc5"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.324270 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.345227 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.349171 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.349433 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.368048 23138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44345
I20260927 18:37:58.368067 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44345 every 8 connection(s)
I20260927 18:37:58.369351 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:58.373063 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.374800 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 23138
I20260927 18:37:58.375017 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-0/wal/instance
I20260927 18:37:58.375456 389 external_mini_cluster.cc:1428] Setting key 04b533f8fc1762348f1a66e5a7ba2029
I20260927 18:37:58.379603 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:58 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
I20260927 18:37:58.401427 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.382645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44345 (local address 127.0.0.1:43872)
0927 18:37:58.383533 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.383582 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.385511 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.386336 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.386352 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.387052 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.387372 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.387389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.388883 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.388891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.390013 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.390025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.390883 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.390889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.391066 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.391714 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.391736 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.394447 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.397581 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.397593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.397604 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.398079 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.398512 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.398516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.398518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.398667 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.399468 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.399474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.399653 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.399954 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.400483 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":50,"thread_start_us":421,"threads_started":1}
I20260927 18:37:58.401516 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.382644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.0.97.126:44345)
0927 18:37:58.384317 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.384326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.384390 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:37:58.385892 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.385897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.386115 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.386335 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.387576 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.388733 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.390144 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.390669 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.390869 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.394664 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.394671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.394682 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.394709 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.397410 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.398222 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.398228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.398230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.398391 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.398767 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.398770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.398773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.399280 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.399463 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.400079 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.400364 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1471,"thread_start_us":1273,"threads_started":1}
I20260927 18:37:58.404538 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:58.405645 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.405265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:58980)
0927 18:37:58.405431 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.405541 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:37:58.405943 23155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
I20260927 18:37:58.407466 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.407134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:58988)
0927 18:37:58.407274 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.407342 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:37:58.407653 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:40589: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260927 18:37:58.431248 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:40589
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:58.457044 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } attempt: 1
I20260927 18:37:58.458463 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.457905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59004)
0927 18:37:58.458145 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.458278 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:37:58.459599 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.459139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59010)
0927 18:37:58.459323 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.459445 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:37:58.459791 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:40589: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
I20260927 18:37:58.528050 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } attempt: 2
I20260927 18:37:58.529408 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.528884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59026)
0927 18:37:58.529107 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.529221 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 18:37:58.530530 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.530117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59036)
0927 18:37:58.530264 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.530373 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 18:37:58.530766 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:40589: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
I20260927 18:37:58.622030 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } attempt: 3
I20260927 18:37:58.623440 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.622783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59052)
0927 18:37:58.623063 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.623206 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:37:58.624706 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.624192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59054)
0927 18:37:58.624390 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.624499 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:37:58.624984 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:40589: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
W20260927 18:37:58.763419 23212 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:58.763778 23212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.763892 23212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.763975 23212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:37:58.768254 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } attempt: 4
I20260927 18:37:58.769524 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.768895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59062)
0927 18:37:58.769241 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.769360 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 18:37:58.770630 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.770237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59064)
0927 18:37:58.770392 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.770489 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:37:58.770840 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:40589: Network error: Client connection negotiation failed: client connection to 127.0.97.125:40589: connect: Connection refused (error 111)
W20260927 18:37:58.772007 23212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:58.772110 23212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.772233 23212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.772305 23212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:58.772392 23212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:58.782539 23212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:58.783886 23212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.785205 23212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:58.796756 23217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.796752 23218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.796859 23220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.797411 23212 server_base.cc:1034] running on GCE node
I20260927 18:37:58.798195 23212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.799485 23212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.800774 23212 hybrid_clock.cc:648] HybridClock initialized: now 1790534278800705 us; error 36 us; skew 500 ppm
I20260927 18:37:58.801167 23212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:58 dist-test-slave-n3p4 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.804574 23212 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260927 18:37:58.806018 23212 webserver.cc:460] Webserver started at http://127.0.97.125:43125/ using document root <none> and password file <none>
I20260927 18:37:58.806690 23212 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.806824 23212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.807138 23212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.810107 23212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data/instance:
uuid: "fed466fe4646445fbba4305151dd470c"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "183361402ee40f8d17f5d2f681fe79b0"
server_key_iv: "2673f27673a9d7c3bb264190618c6bd7"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.810863 23212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal/instance:
uuid: "fed466fe4646445fbba4305151dd470c"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "183361402ee40f8d17f5d2f681fe79b0"
server_key_iv: "2673f27673a9d7c3bb264190618c6bd7"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.815698 23212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 18:37:58.819046 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.820297 23212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:37:58.820482 23212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal
uuid: "fed466fe4646445fbba4305151dd470c"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-n3p4"
server_key: "183361402ee40f8d17f5d2f681fe79b0"
server_key_iv: "2673f27673a9d7c3bb264190618c6bd7"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.820698 23212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.843999 23212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.848693 23212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.849009 23212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.867044 23212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:40589
I20260927 18:37:58.867066 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:40589 every 8 connection(s)
I20260927 18:37:58.868422 23212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/data/info.pb
I20260927 18:37:58.872082 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.875694 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 23212
I20260927 18:37:58.875949 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-1/wal/instance
I20260927 18:37:58.876415 389 external_mini_cluster.cc:1428] Setting key 32194b6a04ce25a73ddff8dcabd4539a
I20260927 18:37:58.878486 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:58 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:58.901998 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.881005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.97.126:44345)
0927 18:37:58.881181 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.881185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.882536 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 18:37:58.884391 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.884397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.884488 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.884599 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.886318 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.887031 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.889459 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.890255 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.890370 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.894737 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.894746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.894754 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.894800 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.898528 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.899316 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.899325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.899332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.899496 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.900011 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.900017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.900024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.900446 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.900607 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.901341 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.901586 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:37:58.902709 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.881587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44345 (local address 127.0.0.1:43874)
0927 18:37:58.882379 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.882416 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.884116 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.884824 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.884832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.885325 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.886089 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.886116 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.887199 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.887207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.889271 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.889289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.890709 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.890717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.891050 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.891785 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.891818 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.894517 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.898702 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.898708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.898720 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.899173 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.899652 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.899656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.899658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.899876 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.900578 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.900584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.900950 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.901167 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.901541 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260927 18:37:58.904495 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
Sep 27 18:37:58 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:58.921940 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.905387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59080)
0927 18:37:58.905605 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.905624 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.905720 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.906990 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.906993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.907335 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.907469 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.907477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.909455 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.909462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.910839 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.910854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.912086 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.912094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.912356 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.913013 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.913037 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.915578 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.918985 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.918993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.919000 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.919530 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.919926 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.919934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.919944 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.920124 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.920736 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.920743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.920920 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.921065 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.921394 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":37}
I20260927 18:37:58.921963 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.905676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.0.97.125:40589)
0927 18:37:58.906438 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.906448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.906481 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:37:58.906552 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.906563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.906822 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.906970 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.907640 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.909310 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.910992 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.911945 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.912091 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.915783 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.915795 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.915807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.915880 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.918847 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.919686 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.919693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.919695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.919808 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.920240 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.920244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.920246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.920605 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.920771 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.921221 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.921507 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":535,"thread_start_us":386,"threads_started":1}
I20260927 18:37:58.924767 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:58.926088 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.925492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49728)
0927 18:37:58.925711 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.925883 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:37:58.926406 23228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:58.928287 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.927890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49730)
0927 18:37:58.928023 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.928121 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:37:58.928530 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260927 18:37:58.939083 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKYOLso/build/asan/bin/kudu
/tmp/dist-test-taskKYOLso/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35567
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:58.977895 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 1
I20260927 18:37:58.979182 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.978620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49740)
0927 18:37:58.978856 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.978994 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 18:37:58.980320 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.979934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49754)
0927 18:37:58.980108 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.980199 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:37:58.980567 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.048835 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 2
I20260927 18:37:59.050086 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.049512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49768)
0927 18:37:59.049756 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.049896 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:37:59.051208 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.050823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49772)
0927 18:37:59.050975 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.051057 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:37:59.051434 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.070071 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } attempt: 5
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:59.084061 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.070898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.0.97.125:40589)
0927 18:37:59.071112 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.071119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.071224 (+ 105us) server_negotiation.cc:407] Connection header received
0927 18:37:59.071444 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.071449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.071537 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.071646 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.072480 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.073189 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.074396 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.074861 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.075000 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.078591 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.078596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.078599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.078634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.081392 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.082147 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.082160 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.082162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.082283 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.082672 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.082681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.082683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.082964 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.083124 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.083535 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.083743 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 18:37:59.084084 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.070805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59094)
0927 18:37:59.071111 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.071130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.071308 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.071687 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.071694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.072142 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.072319 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.072328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.073328 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.073336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.074244 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.074256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.075038 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.075044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.075263 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.075975 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.075995 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.078389 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.081541 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.081548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.081552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.082021 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.082405 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.082410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.082414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.082552 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.083088 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.083101 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.083250 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.083417 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.083720 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260927 18:37:59.085255 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:59.086362 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.085993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49778)
0927 18:37:59.086157 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.086227 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:37:59.087393 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.087001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49790)
0927 18:37:59.087163 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.087266 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:37:59.087602 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.138854 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 1
I20260927 18:37:59.140087 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.139526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49792)
0927 18:37:59.139751 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.139879 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 18:37:59.141085 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.140715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49802)
0927 18:37:59.140869 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.140948 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:37:59.141342 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.142659 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 3
I20260927 18:37:59.143616 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.143172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49808)
0927 18:37:59.143348 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.143440 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:37:59.144727 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.144271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49822)
0927 18:37:59.144514 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.144607 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260927 18:37:59.144963 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.209574 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 2
I20260927 18:37:59.210875 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.210283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49836)
0927 18:37:59.210534 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.210655 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 18:37:59.211954 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.211562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49844)
0927 18:37:59.211721 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.211805 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:37:59.212168 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
W20260927 18:37:59.268905 23286 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:37:59.269217 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:59.269290 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:59.269383 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:59.276827 23286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:59.276912 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:59.276968 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:59.277019 23286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:59.277079 23286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:59.287292 23286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44345,127.0.97.125:40589,127.0.97.124:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:37:59.288185 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 4
I20260927 18:37:59.288789 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:59.289394 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.288869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49858)
0927 18:37:59.289137 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.289218 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:37:59.290418 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.290085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49864)
0927 18:37:59.290223 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.290301 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:37:59.290416 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:59.290657 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
W20260927 18:37:59.302006 23291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.302037 23292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.302067 23294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.303284 23286 server_base.cc:1034] running on GCE node
I20260927 18:37:59.303979 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:59.305253 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:59.306464 23286 hybrid_clock.cc:648] HybridClock initialized: now 1790534279306407 us; error 38 us; skew 500 ppm
I20260927 18:37:59.306743 23286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:59.309859 23286 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260927 18:37:59.311283 23286 webserver.cc:460] Webserver started at http://127.0.97.124:34359/ using document root <none> and password file <none>
I20260927 18:37:59.311924 23286 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:59.312005 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:59.312247 23286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:59.315110 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data/instance:
uuid: "39429878646846659b4d43f14b0fbac4"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-n3p4"
server_key: "011678b31452ec801e64bfc2cc6b4ccd"
server_key_iv: "0ae5d2e024f2c6df4e4a000e4562181f"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.315804 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal/instance:
uuid: "39429878646846659b4d43f14b0fbac4"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-n3p4"
server_key: "011678b31452ec801e64bfc2cc6b4ccd"
server_key_iv: "0ae5d2e024f2c6df4e4a000e4562181f"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.318388 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 3
I20260927 18:37:59.319453 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.318954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49878)
0927 18:37:59.319164 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.319257 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:37:59.320574 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.320173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49890)
0927 18:37:59.320357 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.320428 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 18:37:59.320792 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35567: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35567: connect: Connection refused (error 111)
I20260927 18:37:59.321213 23286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:37:59.324725 23301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.326188 23286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:59.326346 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal
uuid: "39429878646846659b4d43f14b0fbac4"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-n3p4"
server_key: "011678b31452ec801e64bfc2cc6b4ccd"
server_key_iv: "0ae5d2e024f2c6df4e4a000e4562181f"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.326568 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:59.339787 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:59.344029 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:59.344296 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:59.367472 23286 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:35567
I20260927 18:37:59.367439 23352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:35567 every 8 connection(s)
I20260927 18:37:59.369133 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/data/info.pb
I20260927 18:37:59.372849 23353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:59.374413 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKYOLso/build/asan/bin/kudu as pid 23286
I20260927 18:37:59.374611 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKYOLso/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790533919012461-389-0/minicluster-data/master-2/wal/instance
I20260927 18:37:59.375169 389 external_mini_cluster.cc:1428] Setting key 2b3c52993e78c6aa344e95e8e64166e7
I20260927 18:37:59.383100 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:59.401718 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.386331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.97.126:44345)
0927 18:37:59.386569 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.386576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.387320 (+ 744us) server_negotiation.cc:407] Connection header received
0927 18:37:59.387492 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.387498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.387576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.387673 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.388801 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.389817 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.391514 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.392258 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.392385 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.395889 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.395893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.395896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.395954 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.399062 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.399832 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.399835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.399837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.399965 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.400378 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.400383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.400386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.400727 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.400895 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.401310 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.401468 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:37:59.402288 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.386370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44345 (local address 127.0.0.1:43890)
0927 18:37:59.387171 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.387219 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.387362 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.387734 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.387738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.388376 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.388649 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.388657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.389974 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.389980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.391374 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.391385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.392478 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.392483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.392606 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.393209 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.393234 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.395703 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.399182 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.399189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.399194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.399724 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.400069 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.400073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.400078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.400261 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.400874 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.400881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.401034 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.401217 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.401506 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":387,"threads_started":1}
I20260927 18:37:59.402315 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.376986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.0.97.124:35567)
0927 18:37:59.382254 (+ 5268us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.382266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.382308 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:37:59.383903 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.383924 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.384326 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.384711 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.386263 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.388024 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.389585 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.390576 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.390750 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.395281 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.395313 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.395327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.395368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.398032 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.398801 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.398807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.398812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.398970 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.399374 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.399380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.399383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.399975 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.400280 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.400569 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.400903 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5066,"thread_start_us":4875,"threads_started":1}
I20260927 18:37:59.403772 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260927 18:37:59.416448 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.404564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:40589 (local address 127.0.0.1:59096)
0927 18:37:59.404796 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.404810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.404891 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.405246 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.405251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.405611 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.405736 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.405743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.406731 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.406739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.407577 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.407587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.408239 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.408244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.408419 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.408928 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.408943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.411016 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.414096 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.414119 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.414126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.414564 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.415007 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.415010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.415013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.415138 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.415823 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.415826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.415954 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.416058 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.416225 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 18:37:59.416889 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.404489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.97.125:40589)
0927 18:37:59.404685 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.404690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.404885 (+ 195us) server_negotiation.cc:407] Connection header received
0927 18:37:59.405011 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.405015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.405100 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.405216 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.405920 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.406600 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.407706 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.408105 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.408232 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.411189 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.411194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.411197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.411225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.413979 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.414701 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.414709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.414716 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.414889 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.415249 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.415257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.415271 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.415682 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.415847 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.416199 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.416436 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:37:59.417949 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:37:59.420378 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.405318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.97.126:44345)
0927 18:37:59.405679 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.405683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.405695 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:37:59.405729 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.405732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.405822 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.405944 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.407025 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.407711 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.409071 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.409459 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.409607 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.413483 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.413490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.413497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.413534 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.417495 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.418309 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.418313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.418315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.418436 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.418884 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.418889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.418893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.419294 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.419580 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.419801 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.420014 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260927 18:37:59.431531 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.418697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49916)
0927 18:37:59.418908 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.418928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.419030 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.419310 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.419314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.419789 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.419967 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.419980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.421036 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.421041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.422345 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.422360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.423021 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.423025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.423182 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.423754 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.423787 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.426020 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.429145 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.429149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.429153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.429480 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.429923 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.429930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.429934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.430078 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.430552 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.430557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.430710 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.430903 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.431168 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:37:59.431726 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.418900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.0.97.124:35567)
0927 18:37:59.419060 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.419064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.419079 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:37:59.419127 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.419131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.419181 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.419258 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.420114 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.420908 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.422476 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.422860 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.422993 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.426195 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.426199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.426202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.426227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.429013 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.429592 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.429597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.429600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.429714 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.430173 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.430178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.430181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.430438 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.430586 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.431185 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.431400 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:37:59.436146 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: Bootstrap starting.
I20260927 18:37:59.436761 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.422017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.0.97.125:40589)
0927 18:37:59.422188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.422196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.422216 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:37:59.422297 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.422304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.422373 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.422464 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.424418 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.425404 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.426647 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.427065 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.427204 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.431614 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.431620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.431628 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.431676 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.434336 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.434939 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.434951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.434953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.435078 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.435458 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.435463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.435468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.435712 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.435994 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.436257 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.436468 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:59.439962 23353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.441149 23353 log.cc:824] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.441466 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:37:59.444139 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: No bootstrap required, opened a new log
I20260927 18:37:59.444084 23180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e81291caf6204990adb081cee9a987f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900
I20260927 18:37:59.454892 23353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.455292 23353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39429878646846659b4d43f14b0fbac4, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.455789 23353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.456671 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.457051 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.457911 23353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:37:59.467706 23370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:37:59.467849 23370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:37:59.465202 23286 master_runner.cc:186] Error getting master registration for 127.0.97.126:44345: OK, instance_id { permanent_uuid: "e81291caf6204990adb081cee9a987f2" instance_seqno: 1790534278362884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:37:59.476294 23286 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:37:59.498030 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 4
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260927 18:37:59.512590 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.498777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49924)
0927 18:37:59.498970 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.498992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.499124 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.499479 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.499483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.499913 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.500075 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.500084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.501168 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.501176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.502210 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.502222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.503003 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.503009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.503214 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.503845 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.503863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.506190 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.510083 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.510090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.510094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.510462 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.510830 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.510835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.510839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.510975 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.511583 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.511588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.511754 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.511896 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.512201 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:37:59.512607 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.498864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.0.97.124:35567)
0927 18:37:59.499105 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.499110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.499132 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:37:59.499252 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.499258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.499331 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.499444 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.500239 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.500998 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.502353 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.502857 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.502991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.506387 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.506393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.506403 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.506436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.509955 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.510582 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.510588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.510591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.510720 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.511083 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.511089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.511092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.511440 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.511609 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.512015 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.512220 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 18:37:59.517158 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: Bootstrap starting.
I20260927 18:37:59.521190 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.522298 23205 log.cc:824] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.525339 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: No bootstrap required, opened a new log
I20260927 18:37:59.531075 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.531407 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81291caf6204990adb081cee9a987f2, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.531930 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.532881 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.533264 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.534066 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:59.534184 23180 raft_consensus.cc:482] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:37:59.534353 23180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.538476 23180 raft_consensus.cc:504] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.541227 23180 leader_election.cc:288] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [CANDIDATE]: Term 1 election: Requested vote from peers fed466fe4646445fbba4305151dd470c (127.0.97.125:40589), 39429878646846659b4d43f14b0fbac4 (127.0.97.124:35567)
I20260927 18:37:59.541384 23328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e81291caf6204990adb081cee9a987f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39429878646846659b4d43f14b0fbac4"
I20260927 18:37:59.541982 23328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.548110 23328 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81291caf6204990adb081cee9a987f2 in term 1.
I20260927 18:37:59.549007 23154 leader_election.cc:302] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39429878646846659b4d43f14b0fbac4, e81291caf6204990adb081cee9a987f2; no voters:
I20260927 18:37:59.549633 23373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:59.550457 23373 raft_consensus.cc:686] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [term 1 LEADER]: Becoming Leader. State: Replica: e81291caf6204990adb081cee9a987f2, State: Running, Role: LEADER
I20260927 18:37:59.551167 23373 consensus_queue.cc:237] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.555989 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e81291caf6204990adb081cee9a987f2. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.556557 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:59.569674 23138 master_runner.cc:186] Error getting master registration for 127.0.97.125:40589: OK, instance_id { permanent_uuid: "fed466fe4646445fbba4305151dd470c" instance_seqno: 1790534278862116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:37:59.584239 23138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 18:37:59.586617 23390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:37:59.586822 23390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:37:59.587005 23386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:59.589825 23386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:59.589862 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } attempt: 5
W20260927 18:37:59.597282 23307 tablet.cc:2369] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:37:59 dist-test-slave-n3p4 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
W20260927 18:37:59.602240 23159 tablet.cc:2369] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:59.614404 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.591158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.97.124:35567)
0927 18:37:59.591334 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.591339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.591363 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:37:59.591436 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.591442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.591520 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.591616 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.593135 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.594268 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.596070 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.596769 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.596901 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.606696 (+ 9795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.606703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.606708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.606747 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.610774 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.611746 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.611751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.611755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.611911 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.612444 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.612449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.612453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.612789 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.612931 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.613758 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.614022 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:37:59.615197 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.590405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35567 (local address 127.0.0.1:49934)
0927 18:37:59.590769 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.590802 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.590926 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.591848 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.591854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.592404 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.592616 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.592631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.594434 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.594440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.595899 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.595915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.598219 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.598226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.598443 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.599144 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:59.599168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.605699 (+ 6531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.611045 (+ 5346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.611050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.611055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.611599 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.612086 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:59.612093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:59.612098 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:59.612300 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:59.613005 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.613013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.613174 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.613337 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.614757 (+ 1420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260927 18:37:59.619303 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.586092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.97.125:40589)
0927 18:37:59.586290 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.586297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.601520 (+ 15223us) server_negotiation.cc:407] Connection header received
0927 18:37:59.601660 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.601666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.601743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.601941 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.604218 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.605359 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.607655 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.608307 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.608418 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.611933 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.611939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.611944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.611983 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.615655 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.616449 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.616454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.616457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.616622 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.617080 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.617086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.617091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.617458 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.617705 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.618712 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.618963 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:37:59.620569 23328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [term 1 FOLLOWER]: Refusing update from remote peer e81291caf6204990adb081cee9a987f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.622054 23382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.630169 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: Bootstrap starting.
I20260927 18:37:59.632267 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.579779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.0.97.126:44345)
0927 18:37:59.579979 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.579989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.599703 (+ 19714us) server_negotiation.cc:407] Connection header received
0927 18:37:59.602990 (+ 3287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.602998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.603092 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.603168 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.609003 (+ 5835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.610272 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.619649 (+ 9377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.620325 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.620428 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.623948 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.623956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.623963 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.624021 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.627889 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.629671 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.629678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.629683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.629893 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.630413 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.630421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.630425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.630795 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.631051 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.631634 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.631881 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:37:59.634574 23279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.635818 23279 log.cc:824] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.637976 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.603083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.97.124:35567)
0927 18:37:59.608728 (+ 5645us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.608734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.608757 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:59.612280 (+ 3523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.612288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.612374 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.612475 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.615174 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.616439 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.621085 (+ 4646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.621868 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.621989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.625491 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.625499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.625504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:59.625541 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.632819 (+ 7278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.634565 (+ 1746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.634572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.634577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.634830 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:59.635443 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:59.635448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:59.635452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:59.635880 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.636129 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.637260 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.637516 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5509,"thread_start_us":7760,"threads_started":1}
I20260927 18:37:59.645303 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: No bootstrap required, opened a new log
I20260927 18:37:59.656486 23279 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.656868 23279 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed466fe4646445fbba4305151dd470c, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.657378 23279 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } }
I20260927 18:37:59.657763 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e81291caf6204990adb081cee9a987f2. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.658103 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.658993 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.659744 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.660027 23279 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:59.660418 23253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.665071 23253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [term 1 FOLLOWER]: Refusing update from remote peer e81291caf6204990adb081cee9a987f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.665979 23254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e81291caf6204990adb081cee9a987f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fed466fe4646445fbba4305151dd470c"
I20260927 18:37:59.668581 23398 mvcc.cc:205] Tried to move back new op lower bound from 7334028409273147392 to 7334028409047924736. Current Snapshot: MvccSnapshot[applied={T|T < 7334028409273147392}]
I20260927 18:37:59.671120 23254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81291caf6204990adb081cee9a987f2 in term 1.
I20260927 18:37:59.672652 23373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.690218 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.690515 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.698230 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e81291caf6204990adb081cee9a987f2. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.698637 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.700217 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.700870 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:37:59.701011 23386 catalog_manager.cc:1347] Generated new cluster ID: 3f056f8f0f80444bb9d92452f41cb41e
I20260927 18:37:59.701117 23386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:59.702504 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.703080 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:59.703408 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e81291caf6204990adb081cee9a987f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81291caf6204990adb081cee9a987f2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44345 } } peers { permanent_uuid: "fed466fe4646445fbba4305151dd470c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 40589 } } peers { permanent_uuid: "39429878646846659b4d43f14b0fbac4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35567 } } }
I20260927 18:37:59.703959 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:59.722084 23386 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:59.725018 23386 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:59.748791 23386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e81291caf6204990adb081cee9a987f2: Generated new TSK 0
I20260927 18:37:59.750543 23386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:59.788686 23419 catalog_manager.cc:1259] Loaded cluster ID: 3f056f8f0f80444bb9d92452f41cb41e
I20260927 18:37:59.788905 23419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: loading cluster ID for follower catalog manager: success
I20260927 18:37:59.792749 23419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: acquiring CA information for follower catalog manager: success
I20260927 18:37:59.795195 23419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fed466fe4646445fbba4305151dd470c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:37:59.896046 23209 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:59.862499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.97.126:44345)
0927 18:37:59.862682 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.862688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.893808 (+ 31120us) server_negotiation.cc:407] Connection header received
0927 18:37:59.894387 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.894395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.894513 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.894599 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.895771 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43908: BlockingRecv error: recv got EOF from 127.0.0.1:43908 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 18:37:59.898793 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:59.877116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.0.97.125:40589)
0927 18:37:59.877625 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.877633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.896775 (+ 19142us) server_negotiation.cc:407] Connection header received
0927 18:37:59.897262 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.897267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.897367 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.897445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.898571 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59122: BlockingRecv error: recv got EOF from 127.0.0.1:59122 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260927 18:37:59.901744 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:59.899933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.97.124:35567)
0927 18:37:59.900136 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.900142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.900269 (+ 127us) server_negotiation.cc:407] Connection header received
0927 18:37:59.900470 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.900476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.900578 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.900698 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.901498 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49958: BlockingRecv error: recv got EOF from 127.0.0.1:49958 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:37:59.952342 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.931563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.0.97.125:40589)
0927 18:37:59.931760 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.931766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.938276 (+ 6510us) server_negotiation.cc:407] Connection header received
0927 18:37:59.938622 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.938628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.938851 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.938989 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:59.939692 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.940850 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.942964 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.943614 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.943721 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.950606 (+ 6885us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:59.951418 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:59.951568 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.951857 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.951984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 18:37:59.988075 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:59.978697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.97.124:35567)
0927 18:37:59.978886 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.978893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.986367 (+ 7474us) server_negotiation.cc:407] Connection header received
0927 18:37:59.986622 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.986628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.986772 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.986907 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.987797 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49964: BlockingRecv error: recv got EOF from 127.0.0.1:49964 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:38:00.049528 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.025822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.0.97.125:40589)
0927 18:38:00.026021 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.026027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.042361 (+ 16334us) server_negotiation.cc:407] Connection header received
0927 18:38:00.042741 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.042748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.042852 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.042936 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.043652 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.044832 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.046890 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.047547 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.047648 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.048536 (+ 888us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.048700 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.048770 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.049067 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.049156 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 18:38:00.095966 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.090455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.97.124:35567)
0927 18:38:00.090649 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.090656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.094423 (+ 3767us) server_negotiation.cc:407] Connection header received
0927 18:38:00.094613 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.094619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.094727 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.094862 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.095713 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49978: BlockingRecv error: recv got EOF from 127.0.0.1:49978 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:38:00.110618 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.103876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.97.125:40589)
0927 18:38:00.104054 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.104061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.104103 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:38:00.104325 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.104332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.104423 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.104524 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.104999 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.106275 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.108329 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.108993 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.109127 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.109567 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.109732 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.109889 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.110006 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.110093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:38:00.119189 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.116982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.0.97.124:35567)
0927 18:38:00.117173 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.117179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.117694 (+ 515us) server_negotiation.cc:407] Connection header received
0927 18:38:00.117897 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.117903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.118010 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.118114 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.118962 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49992: BlockingRecv error: recv got EOF from 127.0.0.1:49992 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:38:00.134642 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.127704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.0.97.125:40589)
0927 18:38:00.127890 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.127899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.128388 (+ 489us) server_negotiation.cc:407] Connection header received
0927 18:38:00.128552 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.128558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.128638 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.128728 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.129335 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.130439 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.132425 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.133142 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.133271 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.133749 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.133968 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.134059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.134271 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.134354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:38:00.143390 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.141641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.0.97.124:35567)
0927 18:38:00.141871 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.141876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.141896 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:38:00.142049 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.142054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.142147 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.142250 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.143129 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50002: BlockingRecv error: recv got EOF from 127.0.0.1:50002 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:38:00.163374 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.155191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.0.97.125:40589)
0927 18:38:00.155393 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.155400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.155964 (+ 564us) server_negotiation.cc:407] Connection header received
0927 18:38:00.156725 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.156733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.156842 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.156918 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.157603 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.158885 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.161006 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.161856 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.161994 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.162468 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.162647 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.162752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.162955 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.163044 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 18:38:00.173273 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.170994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.124:35567)
0927 18:38:00.171194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.171200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.171762 (+ 562us) server_negotiation.cc:407] Connection header received
0927 18:38:00.171970 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.171976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.172073 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.172180 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.173032 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50014: BlockingRecv error: recv got EOF from 127.0.0.1:50014 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:38:00.200134 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.192785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.97.125:40589)
0927 18:38:00.193021 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.193029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.193573 (+ 544us) server_negotiation.cc:407] Connection header received
0927 18:38:00.193771 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.193831 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.193940 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.194049 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.194623 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.195887 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.197927 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.198645 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.198787 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.199319 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.199473 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.199570 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.199763 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.199861 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 18:38:00.209034 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.206553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.0.97.124:35567)
0927 18:38:00.206731 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.206737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.207288 (+ 551us) server_negotiation.cc:407] Connection header received
0927 18:38:00.207479 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.207485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.207578 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.207683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.208799 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50016: BlockingRecv error: recv got EOF from 127.0.0.1:50016 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:38:00.252770 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.245197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.0.97.125:40589)
0927 18:38:00.245396 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.245403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.246003 (+ 600us) server_negotiation.cc:407] Connection header received
0927 18:38:00.246257 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.246263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.246356 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.246435 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.247130 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.248258 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.250458 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.251205 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.251303 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.251905 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.252050 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.252158 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.252330 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.252437 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:38:00.266295 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.263697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.97.124:35567)
0927 18:38:00.263939 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.263945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.264451 (+ 506us) server_negotiation.cc:407] Connection header received
0927 18:38:00.264745 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.264751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.264859 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.264938 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.266023 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50026: BlockingRecv error: recv got EOF from 127.0.0.1:50026 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:38:00.339012 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.334312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.97.125:40589)
0927 18:38:00.334559 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.334564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.334845 (+ 281us) server_negotiation.cc:407] Connection header received
0927 18:38:00.334997 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.335001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.335071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.335162 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.335641 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.336380 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.337591 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.337990 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.338091 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.338446 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.338541 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.338610 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.338737 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.338799 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 18:38:00.346182 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:00.344317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.0.97.124:35567)
0927 18:38:00.344551 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.344557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.344858 (+ 301us) server_negotiation.cc:407] Connection header received
0927 18:38:00.345055 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.345059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.345127 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.345231 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:00.345999 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50034: BlockingRecv error: recv got EOF from 127.0.0.1:50034 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:38:00.469856 23370 catalog_manager.cc:1259] Loaded cluster ID: 3f056f8f0f80444bb9d92452f41cb41e
I20260927 18:38:00.469954 23370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: loading cluster ID for follower catalog manager: success
I20260927 18:38:00.472314 23370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: acquiring CA information for follower catalog manager: success
I20260927 18:38:00.474010 23370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39429878646846659b4d43f14b0fbac4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:38:00.482666 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.477683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.0.97.125:40589)
0927 18:38:00.477896 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.477900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.478275 (+ 375us) server_negotiation.cc:407] Connection header received
0927 18:38:00.478437 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.478441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.478517 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.478606 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.479060 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.479761 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.481131 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.481532 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.481636 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.482040 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.482150 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.482218 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.482389 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.482448 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:38:00.493614 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.487856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.0.97.124:35567)
0927 18:38:00.488073 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.488081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.488374 (+ 293us) server_negotiation.cc:407] Connection header received
0927 18:38:00.488572 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.488577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.488727 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.488877 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.489246 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.490057 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.491337 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.491726 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.491858 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.492311 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.492787 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.493191 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.493248 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.493329 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:38:00.505592 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.499932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.97.125:40589)
0927 18:38:00.500173 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.500181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.501003 (+ 822us) server_negotiation.cc:407] Connection header received
0927 18:38:00.501188 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.501191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.501256 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.501347 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.501821 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.502527 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.503943 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.504363 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.504508 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.504896 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.504990 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.505085 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.505222 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.505297 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:38:00.511337 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.500125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.0.97.124:35567)
0927 18:38:00.500269 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.500272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.505551 (+ 5279us) server_negotiation.cc:407] Connection header received
0927 18:38:00.505846 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.505852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.505958 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.506030 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.506590 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.507754 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.509649 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.510114 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.510233 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.510584 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.510712 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.510805 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.510925 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.511018 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:38:00.512457 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:00.499586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.97.126:44345)
0927 18:38:00.506219 (+ 6633us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:00.506225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:00.506243 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:38:00.506354 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:00.506362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:00.506622 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:00.506791 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:00.507511 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.508446 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.510279 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:00.510809 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:00.510974 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:00.511447 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:00.511864 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:00.511985 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:00.512071 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 18:38:00.512155 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6513,"thread_start_us":6417,"threads_started":1}
I20260927 18:38:00.516469 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 23138
I20260927 18:38:00.517993 23200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:00.629328 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 23138
I20260927 18:38:00.649955 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 23212
I20260927 18:38:00.651093 23274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:00.758996 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 23212
I20260927 18:38:00.779045 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKYOLso/build/asan/bin/kudu pid 23286
I20260927 18:38:00.780099 23348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:00.887068 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKYOLso/build/asan/bin/kudu with pid 23286
2026-09-27T18:38:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4318 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8658 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361839 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST